ELK 搭建文档

发表于 |

#ELK 搭建

[TOC]

##⚠️

请保持全程elk各系统间的版本一致
elk的数据增长很快，请在开始的时候注意切分好index，方便清理旧数据
本文档仅适用于参考，不同的服务器，版本等信息均需要参看官方教程
一切以官网文档为第一手资料

##安装：

###filebean

####下载

1
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.1.1-x86_64.rpm

####安装

1
rpm -vi filebeat-5.1.1-x86_64.rpm

####配置

1
vi /etc/filebeat/filebeat.yml

####软链

1
ln -s /etc/filebeat /data/local/filebeat

###logstash:

####java环境依赖

1
2
yum install java-1.8.0-openjdk
yum install java-1.8.0-openjdk-devel.x86_64

export JAVACMD=`which java`
export JAVA_HOME=`which java`

####下载

1
2
3
curl -L -O https://artifacts.elastic.co/downloads/logstash/logstash-5.1.1.rpm
sudo rpm -i logstash-5.1.1.rpm

####软链：

1
ln -s /etc/logstash /data/local/logstash

warning: logstash-5.1.1.rpm: Header V4 RSA/SHA512 Signature, key ID d88e42b4: NOKEY

####插件：

plugin 接收插件，gem_source 墙 /usr/share/logstash/Gemfile

filebeat
1
/usr/share/logstash/bin/logstash-plugin install logstash-input-beats

#####start bug fix
/usr/bin/filebeat.sh -e -c /data/local/filebeat/filebeat.yml

##启动

###filebean

####测试

1
/usr/share/filebeat/bin/filebeat -e -c filebeat.yml -d "publish"

1
/usr/share/filebeat/bin/filebeat -e -c /data/local/filebeat/filebeat.yml -d "publish"

####生产

1
nohup /usr/bin/filebeat.sh -e -c /data/local/filebeat/filebeat.yml &>/dev/null &

####清空

测试数据复用

1
rm  /usr/share/filebeat/bin/data/registry

###logstash

####测试

1
2
3
4
/usr/share/logstash/bin/logstash  -f /data/local/logstash/conf.d/logstash.conf
/usr/share/logstash/bin/logstash  -f /etc/logstash/conf.d/logstash.conf
bin/logstash -f first-pipeline.conf --config.test_and_exit
bin/logstash -f first-pipeline.conf --config.reload.automatic

####生产
@(工具)

1
nohup /usr/share/logstash/bin/logstash  -f /data/local/logstash/conf.d/logstash.conf >> /data/

###elasticsearch:

####概览：

  • Check your cluster, node, and index health, status, and statistics
  • Administer your cluster, node, and index data and metadata
  • Perform CRUD (Create, Read, Update, and Delete) and search operations against your indexes
  • Execute advanced search operations such as paging, sorting, filtering, scripting, aggregations, and many others

####下载

1
2
3
4
 
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.1.1.tar.gz
tar -xvf elasticsearch-5.1.1.tar.gz
cd elasticsearch-5.1.1/bin

####启动

1
2
./elasticsearch
./elasticsearch -Ecluster.name=my_cluster_name -Enode.name=my_node_name

####测试
helath check

1
curl -XGET 'localhost:9200/_cat/health?v&pretty'

list of node

1
curl -XGET 'localhost:9200/_cat/nodes?v&pretty'

index of node :

1
curl -XGET 'localhost:9200/_cat/indices?v&pretty'

create index ：

1
curl -XPUT 'localhost:9200/customer?pretty&pretty'

让我们将一个简单的客户文档索引到客户索引“外部”类型，ID为1，如下所示：

新增=修改

1
2
3
4
curl -XPUT 'localhost:9200/customer/external/1?pretty&pretty' -d'
{
  "name": "John Doe"
}'

查看

1
2
curl -XGET 'localhost:9200/customer/external/1?pretty&pretty'
curl -XGET 'localhost:9200/customer/external/1?pretty'

删除

1
curl -XDELETE 'localhost:9200/customer?pretty'

del index

1
curl -XDELETE 'localhost:9200/customer?pretty&pretty'

list index

1
curl -XGET 'localhost:9200/_cat/indices?v&pretty'

本轮存在很大问题啊

1
2
3
4
5
6
7
curl -XPUT 'localhost:9200/customer?pretty'
curl -XPUT 'localhost:9200/customer/external/2?pretty' -d'
{
  "name": "John"
}'
curl -XGET 'localhost:9200/customer/external/1?pretty'
curl -XDELETE 'localhost:9200/customer?pretty'

that in the above case, we are using the POST verb instead of PUT since we didn’t specify an ID.

自增

1
2
3
4
curl -XPOST 'localhost:9200/customer/external?pretty&pretty' -d'
{
  "name": "Jane Doe"
}'

修改1

1
2
3
4
curl -XPOST 'localhost:9200/customer/external/1/_update?pretty&pretty' -d'
{
  "doc": { "name": "Jane Doe", "age": 20 }
}'

修改2

1
2
3
4
curl -XPOST 'localhost:9200/customer/external/1/_update?pretty&pretty' -d'
{
  "script" : "ctx._source.age += 5"
}'

修改 put可以直接进行创建的时候修改

1
curl -XPOST 'localhost:9200/customer/external/1/_update?pretty&pretty' -d'

####批量操作
目前测试不通过，只有批量的第一个会成功

1
2
3
4
5
curl -XPOST 'localhost:9200/customer/external/_bulk?pretty&pretty' -d'
{"index":{"_id":"1"}}
{"name": "John Doe111" }
{"index":{"_id":"2"}}
{"name": "Jane Doe222" }'

####实际数据操作
准备数据

1
2
3
wget  https://raw.githubusercontent.com/elastic/elasticsearch/master/docs/src/test/resources/accounts.json
curl -XPOST 'localhost:9200/bank/account/_bulk?pretty&refresh' --data-binary "@accounts.json"
curl 'localhost:9200/_cat/indices?v'

查询 REST request URI 

1
curl -XGET 'localhost:9200/bank/_search?q=*&sort=account_number:asc&pretty&pretty'

查询 REST request body 

1
2
3
4
5
6
7
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "sort": [
    { "account_number": "asc" }
  ]
}'

查询扩展 REST request body
size 默认 10

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
查询所有
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_all": {} }
}'
查询所有 返回1
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "size": 1
}'
查询所有 返回10条
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "from": 10,
  "size": 10
}'
查询所有 返回字段仅 account_number balance
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_all": {} },
  "_source": ["account_number", "balance"]
}'
查询匹配 account_number = 20
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match": { "account_number": 20 } }
}'
查询匹配 
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match": { "address": "mill" } }
}'
查询匹配 or
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match": { "address": "mill lane" } }
}'
查询匹配 &&
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": { "match_phrase": { "address": "mill lane" } }
}'
查询匹配 &&
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}'
查询 ！
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": {
    "bool": {
      "must_not": [
        { "match": { "address": "mill" } },
        { "match": { "address": "lane" } }
      ]
    }
  }
}'
组合
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": {
    "bool": {
      "must": [
        { "match": { "age": "40" } }
      ],
      "must_not": [
        { "match": { "state": "ID" } }
      ]
    }
  }
}'
范围
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "query": {
    "bool": {
      "must": { "match_all": {} },
      "filter": {
        "range": {
          "balance": {
            "gte": 20000,
            "lte": 30000
          }
        }
      }
    }
  }
}'
分组查询
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      }
    }
  }
}'
＝＝
SELECT state, COUNT(*) FROM bank GROUP BY state ORDER BY COUNT(*) DESC
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword",
        "order": {
          "average_balance": "desc"
        }
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}'
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "size": 0,
  "aggs": {
    "group_by_state": {
      "terms": {
        "field": "state.keyword"
      },
      "aggs": {
        "average_balance": {
          "avg": {
            "field": "balance"
          }
        }
      }
    }
  }
}'
这个例子演示了我们如何根据年龄段（20-29，30-39和40-49），然后按性别分组，然后最终得到每个年龄段的每个性别的平均帐户余额！！！
curl -XGET 'localhost:9200/bank/_search?pretty' -d'
{
  "size": 0,
  "aggs": {
    "group_by_age": {
      "range": {
        "field": "age",
        "ranges": [
          {
            "from": 20,
            "to": 30
          },
          {
            "from": 30,
            "to": 40
          },
          {
            "from": 40,
            "to": 50
          }
        ]
      },
      "aggs": {
        "group_by_gender": {
          "terms": {
            "field": "gender.keyword"
          },
          "aggs": {
            "average_balance": {
              "avg": {
                "field": "balance"
              }
            }
          }
        }
      }
    }
  }
}'

####elastic search 扩展练习

1
2
3
4
curl -XGET '127.0.0.1:9200/xmpp/_search?pretty' -d'
{
  "query": { "match": { "message": "1000010=>1279133" } }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
GET /megacorp/employee/_search?pretty
GET /megacorp/employee/_search?q=last_name:Smith
GET /megacorp/employee/_search?pretty
{
    "query" : {
        "match" : {
            "last_name" : "Smith"
        }
    }
}
GET /megacorp/employee/_search?pretty
{
    "query" : {
        "bool" : {
            "must" : {
                "match" : {
                    "last_name" : "smith" 
                }
            },
            "filter" : {
                "range" : {
                    "age" : { "gt" : 30 } 
                }
            }
        }
    }
}
GET /megacorp/employee/_search
{
    "query" : {
        "match" : {
            "about" : "rock climbing"
        }
    }
}
GET /megacorp/employee/_search
{
    "query" : {
        "match_phrase" : {
            "about" : "rock climbing"
        }
    }
}
GET /megacorp/employee/_search
{
    "query" : {
        "match_phrase" : {
            "about" : "rock climbing"
        }
    },
    "highlight": {
        "fields" : {
            "about" : {}
        }
    }
}
GET /megacorp/employee/_search
{
  "aggs": {
    "all_interests": {
      "terms": { "field": "interests" }
    }
  }
}
PUT /megacorp/_mapping/employee?pretty
{
  "properties": {
    "interests": {
      "type":     "text",
      "fielddata": true
    }
  }
}
GET /megacorp/employee/_search
{
  "query": {
    "match": {
      "last_name": "smith"
    }
  },
  "aggs": {
    "all_interests": {
      "terms": {
        "field": "interests"
      }
    }
  }
}
GET /megacorp/employee/_search
{
    "aggs" : {
        "all_interests" : {
            "terms" : { "field" : "interests" },
            "aggs" : {
                "avg_age" : {
                    "avg" : { "field" : "age" }
                }
            }
        }
    }
}

###kibana

####安装
vi /etc/yum.repos.d/kibana.repo

1
2
3
4
5
6
7
8
9
[kibana-5.x]
name=Kibana repository for 5.x packages
baseurl=https://artifacts.elastic.co/packages/5.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

1
2
3
4
5
sudo yum install kibana
sudo chkconfig --add kibana
sudo -i service kibana start
sudo -i service kibana stop

####运行
sudo -i service kibana start

####配置
nginx转发

1
sudo -i service kibana stop

####elastic search生产环境部署bug fix：

#####Q1:
ERROR: bootstrap checks failed system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

1
2
3
4
5
6
7
原因：
这是在因为Centos6不支持SecComp，而ES5.2.0默认bootstrap.system_call_filter为true进行检测，所以导致检测失败，失败后直接导致ES不能启动。
解决：
在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false`

#####Q2:
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]

1
2
3
4
5
6
7
8
9
10
11
12
13
`ulimit -n 5000`
解决：切换到root用户，进入limits.d目录下修改配置文件。
vi /etc/security/limits.d/90-nproc.conf 
修改如下内容：
* soft nproc 1024
#修改为
* soft nproc 2048

#####Q3
max number of threads [1024] for user [lishang] likely too low, increase to at least [2048]

1
2
3
4
5
6
7
8
9
10
11
12
13
解决：切换到root用户修改配置sysctl.conf
vi /etc/sysctl.conf 
添加下面配置：
vm.max_map_count=655360
并执行命令：
sysctl -p
然后，重新启动elasticsearch，即可启动成功。

###x-pack

x-pack 是需要license的，请安装的时候注意，如果不准备申请license的话：
配置如下：
elasticsearch.yml

1
2
3
4
xpack.security.enabled: false
xpack.monitoring.enabled: true
xpack.graph.enabled: false
#xpack.reporting.enabled: false

1
2
3
some warm：
Storing generated key in [/Users/langlive/Desktop/elasticsearch-5.2.1/config/x-pack/system_key]...
Ensure the generated key can be read by the user that Elasticsearch runs as, permissions are set to owner read/write only

####安装
elasticsearch

1
bin/elasticsearch-plugin install x-pack

kibana

1
bin/kibana-plugin install x-pack

logstash

1
bin/logstash-plugin install x-pack

filebeat

1
vim /data/local/filebeat/filebeat.yml

####设置

#####密码

1
2
3
4
5
6
7
8
9
10
11
curl -XPUT -u elastic '127.0.0.1:9200/_xpack/security/user/elastic/_password' -d '{
  "password" : "123456"
}'
curl -XPUT -u elastic '127.0.0.1:9200/_xpack/security/user/kibana/_password' -d '{
  "password" : "123456"
}'
curl -XPUT -u elastic '127.0.0.1:9200/_xpack/security/user/logstash_system/_password' -d '{
  "password" : "123456"
}'

#####访问控制

######创建角色

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
curl -XPOST -u elastic '127.0.0.1:9200/_xpack/security/role/events_admin' -d '{
  "indices" : [
    {
      "names" : [ "events*" ],
      "privileges" : [ "all" ]
    },
	{
      "names" : [ "xmpp*" ],
      "privileges" : [ "all" ]
    },
	{
      "names" : [ "log*" ],
      "privileges" : [ "all" ]
    },
    {
      "names" : [ ".kibana*" ],
      "privileges" : [ "manage", "read", "index" ]
    }
  ]
}'
curl -XPOST -u elastic '127.0.0.1:9200/_xpack/security/role/events_root' -d '{
  "cluster":all
  "indices" : [
    {
      "names" : [ "*" ],
      "privileges" : [ "all" ]
    },
  ]
}'

######创建管理员

1
2
3
4
5
6
7
8
9
10
11
12
13
14
curl -XPOST -u elastic '127.0.0.1:9200/_xpack/security/user/walter' -d '{
  "password" : "enjoyprocess",
  "full_name" : "walter.shi",
  "email" : "walter.shi@langlive.com",
  "roles" : [ "events_admin" ]
}'
curl -XPOST -u elastic '127.0.0.1:9200/_xpack/security/user/dev' -d '{
  "password" : "kibana-search",
  "full_name" : "langlive.dev",
  "email" : "dev@langlive.com",
  "roles" : [ "events_admin" ]
}'

#####修改密码

1
2
3
4
5
6
curl -XPUT 'localhost:9200/_xpack/security/user/elastic/_password?pretty' -H 'Content-Type: application/json' -d'
{
  "password": "elasticpassword"
}
'

参考链接：
filebeat安装文档教程
filebeat配置文档教程
logstash安装文档教程
logstash安装插件教程
logstash配置文档教程