分类 WEB开发 下的文章

Elastichsearch数据备份,恢复,及迁移

目的:将数据从A集群迁移到B集群,大概步骤:创建快照->导出->迁移备份->导入

1、vi /etc/elasticsearch/elasticseach.yml, 添加path.repo: /path/to/backup,重启A集群的Elasticseach. B集群也要执行此步骤,且所有的节点都需要。

2、创建快照(B集群也需要)

curl -XPUT  http://127.0.0.1:9500/_snapshot/my_backup -d '
{
    "type": "fs",
    "settings": {
        "location": "/path/to/backup",
        "compress" : true
    }
}
'

curl -XGET http://127.0.0.1:9500/_snapshot/my_backup //检查快照信息

3、导出备份文件

curl -XPUT '127.0.0.1:9500/_snapshot/my_backup/snapshot_20170116?pretty' -d'
{
  "indices": "poi.v1,poi.v2", //要到处的索引
  "ignore_unavailable": true,
  "include_global_state": false
}
'

curl -XGET http://127.0.0.1:9500/_snapshot/my_backup/snapshot_20170116/_status //查看导出备份状态,DONE标示已经完成

curl -XDELETE http://127.0.01:9500/_snapshot/my_backup/snapshot_20170116 //删除正在导出的备份(取消备份)

4、迁移备份

scp || sync A -> B (B集群中的所有节点都需要同步)

5、恢复备份(导入)

curl -XPOST http://127.0.0.1:9200/_snapshot/my_backup/snapshot_20170116/_restore?wait_for_completion=true

es-name-test.sh

#!/bin/sh
# Delete old version to make sure new settings are applied
curl -XDELETE "localhost:9200/names-test/"
echo
# See https://github.com/elasticsearch/elasticsearch-analysis-phonetic
curl -XPUT "localhost:9200/names-test" -d '{
    "settings": {
        "analysis": {
            "analyzer": {
                "lowkeyword-analyzer": {
                    "type": "custom",
                    "tokenizer": "keyword",
                    "filter" : ["lowercase"]
                },
                "phonetic-analyzer": {
                    "type": "custom",
                    "tokenizer": "whitespace",
                    "filter" : ["standard", "lowercase", "dmeta-filter"]
                },
                "nickname-analyzer": {
                    "type": "custom",
                    "tokenizer": "whitespace",
                    "filter" : ["onetoken-filter", "lowercase", "nickname-filter"]
                }
            },
            "filter" : {
                "dmeta-filter" : {
                    "type" : "phonetic",
                    "encoder" : "doublemetaphone",
                    "replace" : true
                },
                "onetoken-filter" : {
                    "type" : "limit",
                    "max_token_count" : 1
                },
                "nickname-filter" : {
                    "type" : "synonym",
                    "synonyms_path" : "../config/analysis/nick2basename.txt"
                }
            }
        }
    },
    "mappings": {
        "_default_": {
          "_timestamp" : {
            "enabled" : true,
            "store" : true
          }
        },
        "doc": {
            "properties": {
                "name" : {
                  "type" : "multi_field",
                  "fields" : {
                    "name" : {"type" : "string", "index" : "analyzed",
                        "index_analyzer": "standard", "search_analyzer": "standard"},
                    "asis" : {"type" : "string", "index" : "analyzed",
                        "index_analyzer": "lowkeyword-analyzer", "search_analyzer": "lowkeyword-analyzer"},
                    "phonetic" : {"type" : "string", "index" : "analyzed",
                        "index_analyzer": "phonetic-analyzer", "search_analyzer": "phonetic-analyzer"},
                    "basename" : {"type" : "string", "index" : "analyzed",
                        "index_analyzer": "nickname-analyzer", "search_analyzer": "nickname-analyzer"}
                  }
                },
                "date" : {
                  "type" : "date",
                  "format" : "dateOptionalTime"
                }
            }
        }
    }
}'


# curious about path analyzer? test it:
echo testing lowkeyword analyzier
curl -XGET 'localhost:9200/names-test/_analyze?analyzer=lowkeyword-analyzer&pretty=1' -d 'ALAN JURGENSEN'
echo

echo testing phonetic analyzier
curl -XGET 'localhost:9200/names-test/_analyze?analyzer=phonetic-analyzer&pretty=1' -d 'ALEN JORGENSEN'
echo

echo testing nickname analyzier
curl -XGET 'localhost:9200/names-test/_analyze?analyzer=nickname-analyzer&pretty=1' -d 'BILL ALAN'
echo

#!/bin/sh
# first load :  schema-accts-search-exp.sh

putdata () {
# Put some new docs
curl -sXPUT "localhost:9200/names-test/doc/1001" -d '{"name": "Joe Johnson", "date": "2013-08-01"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1002" -d '{"name": "JOHN JOHNSON", "date": "2013-08-02"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1003" -d '{"name": "jeff smith", "date": "2013-08-03"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1004" -d '{"name": "Jeff Johnson", "date": "2013-08-04"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1005" -d '{"name": "john smith", "date": "2013-08-05"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1006" -d '{"name": "ALAN HANSON", "date": "2013-08-06"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1007" -d '{"name": "MIKE SMITH", "date": "2013-08-07"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1008" -d '{"name": "ALAN JURGENSEN", "date": "2013-08-08"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1009" -d '{"name": "ALLEN JORGENSON", "date": "2013-08-09"}'
echo
curl -sXPUT "localhost:9200/names-test/doc/1010" -d '{"name": "ALLEN JURGENSON", "date": "2013-08-10"}'
echo

curl -sXPOST "localhost:9200/names-test/_refresh"
echo
# check for existence via:
curl -sXHEAD 'localhost:9200/names-test/doc/1001'
echo
# get doc
curl -sXGET 'localhost:9200/names-test/doc/1008?pretty'
echo
curl -sXGET 'localhost:9200/names-test/doc/1009?pretty'
echo

# get specific fields (default field is _source)
curl -sXGET 'localhost:9200/names-test/doc/1008?fields=_timestamp,name&pretty'
echo
}

# search and match _all
curl -sXPOST "localhost:9200/names-test/_search?pretty" -d '{"fields": ["_timestamp","_source"], "query": {"match_all" : {}}}'
echo

# curious about path analyzer? test it:
echo testing lowkeyword analyzier
curl -sXGET 'localhost:9200/names-test/_analyze?analyzer=lowkeyword-analyzer&pretty' -d 'ALAN JURGENSEN'
echo

echo testing phonetic analyzier
curl -sXGET 'localhost:9200/names-test/_analyze?analyzer=phonetic-analyzer&pretty' -d 'ALAN JURGENSEN'
echo

echo testing nickname to basename analyzier
curl -sXGET 'localhost:9200/names-test/_analyze?analyzer=nickname-analyzer&pretty' -d 'BILL ALAN'
echo

echo 'Name match query: name'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "match": {
            "name": "ALAN"
        }
    }
}'
echo

echo 'Name match query: name.asis'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "match": {
            "name.asis": "ALAN"
        }
    }
}'
echo

echo 'Name match query: name.asis'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "match": {
            "name.asis": "alan jurgensen"
        }
    }
}'
echo

echo 'Name match query: name.phonetic'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "match": {
            "name.phonetic": "alan jurgensen"
        }
    }
}'
echo

echo 'Name match query: name.basename'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "match": {
            "name.basename": "allen"
        }
    }
}'
echo

echo 'Name multi-match query: name.name name.asis name.phonetic name.basename'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "multi_match": {
            "query": "alan Jurgensen",
            "fields": [ "name.name", "name.asis", "name.phonetic", "name.basename" ]
        }
    }
}'
echo

echo 'Name multi-match BOOST query: name.name name.asis name.phonetic name.basename'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty" -d '{
    "query": {
        "multi_match": {
            "query": "alan Jurgensen",
            "fields": [ "name.name^1.6", "name.asis^3", "name.phonetic^1.4", "name.basename^1.2" ]
        }
    }
}'
echo

echo 'Name multi-match BIGBOOST query: name.name name.asis name.phonetic name.basename'
curl -sXPOST "localhost:9200/names-test/doc/_search?pretty&explain" -d '{
    "query": {
            "multi_match": {
                "query": "alan Jurgensen",
                "fields": [ "name.name^4", "name.asis^6", "name.phonetic^3", "name.basename^2" ]
            }
    }
}'
echo

sublime text3 user setting

{
    "always_show_minimap_viewport": true,
    "bold_folder_labels": true,
    "caret_extra_bottom": 1,
    "caret_extra_top": 1,
    "caret_extra_width": 1,
    "caret_style": "blink",
    "color_scheme": "Packages/Boxy Theme/schemes/Boxy Tomorrow.tmTheme",
    "default_line_ending": "LF",
    "enable_tab_scrolling": false,
    "fade_fold_buttons": false,
    "folder_exclude_patterns":
    [
        ".svn",
        ".git",
        ".hg",
        "CVS",
        ".phpintel",
        ".idea"
    ],
    "font_size": 14,
    "highlight_line": true,
    "ignored_packages":
    [
        "Vintage"
    ],
    "indent_guide_options":
    [
        "draw_normal",
        "draw_active"
    ],
    "line_padding_bottom": 2,
    "line_padding_top": 2,
    "overlay_scroll_bars": "enabled",
    "rulers":
    [
        80
    ],
    "show_encoding": true,
    "show_line_endings": true,
    "tab_size": 4,
    "theme": "Boxy Tomorrow.sublime-theme",
    "theme_accent_green": true,
    "theme_bar": true,
    "theme_find_panel_close_hidden": true,
    "theme_sidebar_size_sm": true,
    "theme_size_lg": true,
    "theme_statusbar_size_md": true,
    "theme_tab_line_size_lg": true,
    "theme_tab_selected_transparent": true,
    "theme_tab_selected_underlined": true,
    "theme_tab_separator": true,
    "translate_tabs_to_spaces": true,
    "word_wrap": false,
    "wrap_width": 100
}

location 匹配规则(转)

location 匹配规则

语法规则

location [=|~|~*|^~] /uri/ { … }

符号 含义
= 开头表示精确匹配
^~ 开头表示uri以某个常规字符串开头,理解为匹配 url路径即可。nginx不对url做编码,因此请求为/static/20%/aa,可以被规则^~ /static/ /aa匹配到(注意是空格)
~ 开头表示区分大小写的正则匹配
~* 开头表示不区分大小写的正则匹配
!~和!~* 分别为区分大小写不匹配及不区分大小写不匹配 的正则
/ 通用匹配,任何请求都会匹配到。

多个location配置的情况下匹配顺序为(参考资料而来,还未实际验证,试试就知道了,不必拘泥,仅供参考):

  • 首先匹配 =
  • 其次匹配 ^~
  • 其次是按文件中顺序的正则匹配
  • 最后是交给 / 通用匹配
  • 当有匹配成功时候,停止匹配,按当前匹配规则处理请求

例子,有如下匹配规则:

location = / {
   #规则A
}
location = /login {
   #规则B
}
location ^~ /static/ {
   #规则C
}
location ~ \.(gif|jpg|png|js|css)$ {
   #规则D
}
location ~* \.png$ {
   #规则E
}
location !~ \.xhtml$ {
   #规则F
}
location !~* \.xhtml$ {
   #规则G
}
location / {
   #规则H
}

那么产生的效果如下:

访问 http://localhost/category/id/1111 则最终匹配到规则H,因为以上规则都不匹配,这个时候应该是nginx转发请求给后端应用服务器,比如FastCGI(php),tomcat(jsp),nginx作为方向代理服务器存在。

所以实际使用中,个人觉得至少有三个匹配规则定义,如下:

#直接匹配网站根,通过域名访问网站首页比较频繁,使用这个会加速处理,官网如是说。
#这里是直接转发给后端应用服务器了,也可以是一个静态首页
# 第一个必选规则
location = / {
    proxy_pass http://tomcat:8080/index
}
# 第二个必选规则是处理静态文件请求,这是nginx作为http服务器的强项
# 有两种配置模式,目录匹配或后缀匹配,任选其一或搭配使用
location ^~ /static/ {
    root /webroot/static/;
}
location ~* \.(gif|jpg|jpeg|png|css|js|ico)$ {
    root /webroot/res/;
}
#第三个规则就是通用规则,用来转发动态请求到后端应用服务器
#非静态文件请求就默认是动态请求,自己根据实际把握
#毕竟目前的一些框架的流行,带.php,.jsp后缀的情况很少了
location / {
    proxy_pass http://tomcat:8080/
}

ReWrite语法

  • last – 基本上都用这个Flag
  • break – 中止Rewirte,不在继续匹配
  • redirect – 返回临时重定向的HTTP状态302
  • permanent – 返回永久重定向的HTTP状态301

1、下面是可以用来判断的表达式:

-f和!-f用来判断是否存在文件
-d和!-d用来判断是否存在目录
-e和!-e用来判断是否存在文件或目录
-x和!-x用来判断文件是否可执行

2、下面是可以用作判断的全局变量

例:http://localhost:88/test1/test2/test.php
$host:localhost
$server_port:88
$request_uri:http://localhost:88/test1/test2/test.php
$document_uri:/test1/test2/test.php
$document_root:D:\nginx/html
$request_filename:D:\nginx/html/test1/test2/test.php

Redirect语法

server {
    listen 80;
    server_name start.igrow.cn;
    index index.html index.php;
    root html;
    if ($http_host !~ “^star\.igrow\.cn$&quot {
        rewrite ^(.*) http://star.igrow.cn$1 redirect;
    }
}

防盗链

location ~* \.(gif|jpg|swf)$ {
    valid_referers none blocked start.igrow.cn sta.igrow.cn;
    if ($invalid_referer) {
       rewrite ^/ http://$host/logo.png;
    }
}

根据文件类型设置过期时间

location ~* \.(js|css|jpg|jpeg|gif|png|swf)$ {
    if (-f $request_filename) {
        expires 1h;
        break;
    }
}

禁止访问某个目录

location ~* \.(txt|doc)${
    root /data/www/wwwroot/linuxtone/test;
    deny all;
}