ELK E Elastic Search L Logstash K Kibana

什么是ELK E: Elastic. Search L: Logstash K: Kibana

环境搭建(一) Elastic. Search安装 1、安装elasticsearch的yum源的密钥 rpm --import https: //artifacts. elastic. co/GPG-KEY-elasticsearch 2、配置elasticsearch的yum源 vim /etc/yum. repos. d/elasticsearch. repo [elasticsearch-6. x] name=Elasticsearch repository for 6. x packages baseurl=https: //artifacts. elastic. co/packages/6. x/yum gpgcheck=1 gpgkey=https: //artifacts. elastic. co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md 3、安装elasticsearch yum install -y elasticsearch

环境搭建(一) Elastic. Search环境搭建 1、需要安装jdk 1. 8版本以上的 java -version 2、创建elasticsearch data的存放目录，并修改该目录的属主属组 mkdir -p /data/es-data chown -R elasticsearch: elasticsearch /data/es-data 3、修改elasticsearch的日志属主属组 chown -R elasticsearch: elasticsearch /var/log/elasticsearch/ 4、修改elasticsearch的配置文件 vim /etc/elasticsearch. yml

/etc/elasticsearch. yml编 辑 找到配置文件中的cluster. name，打开该配 置并设置集群名称 cluster. name: elk-tang 找到配置文件中的node. name，打开该配置 并设置节点名称 node. name: elk-tang-1 修改data存放的路径 path. data: /data/es-data 修改logs日志的路径 path. logs: /var/log/elasticsearch/ 注释配置内存使用用交换分区 #bootstrap. memory_lock: true 监听的网络地址 network. host: 0. 0 开启监听的端口 http. port: 9200 增加新的参数，这样head插件可以访问es (5. x 版本，如果没有可以自己手动加) http. cors. enabled: true http. cors. allow-origin: "*"

启动Elastic. Search /etc/init. d/elasticsearch start

创建开机自启动服务 chkconfig elasticsearch on

其他需要修改的参数 vim /etc/security/limits. conf 在末尾追加以下内容（elk为启动用户，当然也可以指定为*） elk soft nofile 65536 elk hard nofile 65536 elk soft nproc 2048 elk hard nproc 2048 elk soft memlock unlimited elk hard memlock unlimited vim /etc/security/limits. d/XXX-nproc. conf 将里面的1024改为 2048（ES最少要 求为 2048） * soft nproc 2048 vim /etc/elasticsearch. yml加入以下内容 bootstrap. system_call_filter: false

再次启动 /etc/init. d/elasticsearch restart

环境搭建（二） 安装elasticsearch-head插件 安装node. js sudo curl -s. L -o /etc/yum. repos. d/khara-nodejs. repo https: //copr. fedoraproject. org/coprs/khara/nodejs/repo/epel-7/kharanodejs-epel-7. repo sudo yum install -y nodejs-npm 安装head git clone git: //github. com/mobz/elasticsearch-head. git cd elasticsearch-head npm install npm run start

环境搭建（三） 安装Logstash环境 Logstash需要安装到产生日志的服务器上 rpm --import https: //artifacts. elastic. co/GPG-KEY-elasticsearch yum install -y logstash rpm -ql logstash ln -s /usr/share/logstash/bin/logstash /bin/

Logstash配置 input { file { path => [ "/var/log/nginx/access. log" ] start_position => "beginning" ignore_older => 0 } mutate { convert => [ "[geoip][coordinates]", "float" ] convert => [ "response", "integer" ] convert => [ "bytes", "integer" ] replace => { "type" => "nginx_access" } remove_field => "message" } } date { match => [ "timestamp", "dd/MMM/yyyy: HH: mm: ss Z"] filter { } mutate { remove_field => "timestamp" grok { patterns_dir => " /opt/logstash/patterns" match => { "message" => "%{NGINXACCESS}" } } add_field => [ “resp_code”, “%{response}” ] } geoip { source => "http_x_forwarded_for" target => "geoip" database => "/etc/logstash/Geo. Lite 2 -City. mmdb" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } } output { elasticsearch { hosts => ["127. 0. 0. 1: 9200"] index => "logstash-nginx-access-%{+YYYY. MM. dd}" } stdout {codec => rubydebug} }

建立grok使用的表达式 mkdir -pv /opt/logstash/patterns vi /opt/logstash/patterns/nginx NGUSERNAME [a-z. A-Z. @-+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST: clientip} - %{NOTSPACE: remote_user} [%{HTTPDATE: timestamp}] "(? : %{WORD: verb} %{NOTSPACE: request} HTTP/%{NUMBER: httpversion}|%{DATA: rawrequest})" %{NUMBER: response} (? : %{NUMBER: bytes}|-) %{QS: referrer} %{QS: agent} "(? : %{IPV 4: http_x_forwarded_for}|-)"

Geo. IP的数据库解析ip wget http: //geolite. maxmind. com/download/geoip/database/Geo. Lite 2 City. tar. gz tar -xzvf Geo. Lite 2 -City. tar. gz mv Geo. Lite 2 -City_20181030/Geo. Lite 2 -City. mmdb /etc/logstash/.

测试配置文件并启动Logstash服务 logstash -t -f. /elk. conf nohup logstash –f. /elk. conf 2>&1 > /dev/null &

环境搭建（四） Kibana wget https: //artifacts. elastic. co/downloads/kibana-6. 4. 2 -linuxx 86_64. tar. gz # 注意需要与ES对应的版本 tar -xzf kibana-6. 4. 2 -linux-x 86_64. tar. gz mv kibana-6. 4. 2 -linux-x 86_64 /usr/local ln -s /usr/local/kibana-6. 4. 2 -linux-x 86_64/ /usr/local/kibana vim /usr/local/kibana/config/kibana. yml

/usr/local/kibana/config/kibana. yml编 辑 server. port: 5601 server. host: "0. 0" elasticsearch. url: "http: //localhost: 9200" kibana. index: ". kibana"

安装screen, 以便于kibana在后台运行 yum -y install screen /usr/local/kibana/bin/kibana

Kibana安装完成 打开浏览器并设置对应的index http: //localhost: 5601

在Kibana上安装sentinl插件用于发送邮件 提醒 1、到https: //github. com/sirensolutions/sentinl/releases上选择合适的版本 2、在服务器上运行： /usr/local/kibana/bin/kibana-plugin install https: //github. com/sirensolutions/sentinl/releases/download/tag-6. 4. 20/sentinl-v 6. 4. 2. zip 3、重启kibana 4、在kibana界面上配置sentinl

注意：需要定时清理日志文件 定期清理nginx日志文件 echo ': : 1 - - [03/Nov/2018: 20: 28: 03 +0800] "GET / HTTP/1. 1" 200 0 "-" "Mozilla/5. 0 (X 11; Linux x 86_64; rv: 52. 0) Gecko/20100101 Firefox/52. 0" "-"' > /var/log/nginx/access. log 定期清理ES中的日志文件 curl -X DELETE http: //localhost: 9200/nginx-*-`date +%Y-%m-%d -d "-$n days"`