Elasticsearch+Logstash+Kibana实战部署[ELK]

Elasticsearch部署

一、解压elasticsearch并配置bin/elasticsearch.yml

[root@localhost tmp]# tar -zxvf elasticsearch-7.4.3.tar.gz
[root@localhost tmp]# mkdir /ELK
[root@localhost tmp]# mv elasticsearch-7.4.3 /ELK/elasticsearch
[root@localhost /]# cd ELK/elasticsearch/config/
[root@localhost config]# vim elasticsearch.yml


配置如下：

cluster.name: ELK-Cluster #ELK的集群名称，名称相同即属于是同一个集群
node.name: elk-node1 #本机在集群内的节点名称
path.data: /ELK/elasticsearch/data #数据存放目录
path.logs: /ELK/elasticsearch/data/log #日志保存目录
bootstrap.memory_lock: true #服务启动的时候锁定足够的内存，防止数据写入swap
network.host: 172.18.12.164 #监听的IP地址
http.port: 9200 #服务监听的端口
discovery.seed_hosts: ["172.18.12.172", "172.18.12.173"] #单播配置一台即可
cluster.initial_master_nodes: ["elk-node1", "elk-node2"]


二、配置内存限制

[root@localhost config]# vim jvm.options


配置如下：

-Xms2g
-Xmx2g


最小和最大内存限制，为什么最小和最大设置一样大？
参考：https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html

三、创建用户以及数据目录

[root@localhost config]# mkdir -p /ELK/elasticsearch/{data,data/log}
[root@localhost elasticsearch]# groupadd elk
[root@localhost elasticsearch]# useradd -g elk elk
[root@localhost elasticsearch]# passwd elk
[root@localhost elasticsearch]# chown -R elk:elk /ELK/elasticsearch/ -R
[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch


ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
[2]: memory locking requested for elasticsearch process but memory is not locked
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]


启动报错：需要修改文件的最大打开数以及内存锁定

修改内核参数

[root@localhost bin]# vim /etc/security/limits.conf


配置如下：

* soft nofile 65536
* hard nofile 131072
* soft memlock unlimited
* hard memlock unlimited


修改最大线程数

[root@localhost bin]# vim /etc/security/limits.d/20-nproc.conf


配置如下：

* soft nproc unlimited


[root@localhost bin]# vim /etc/sysctl.conf


配置如下：

vm.max_map_count = 655360


令配置生效：

[root@localhost bin]# sysctl -p
[root@localhost bin]# reboot


四、启动服务

[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch
[elk@localhost bin]$ yum -y install net-tools
[elk@localhost bin]# netstat -nltup |grep java


tcp6 0 0 172.18.12.164:9200 :::* LISTEN 9971/java
tcp6 0 0 172.18.12.164:9300 :::* LISTEN 9971/java


同理配置另一台bin/elasticsearch.yml

[root@localhost tmp]# tar -zxvf elasticsearch-7.4.3.tar.gz
[root@localhost tmp]# mkdir /ELK
[root@localhost tmp]# mv elasticsearch-7.4.3 /ELK/elasticsearch
[root@localhost /]# cd ELK/elasticsearch/config/
[root@localhost config]# vim elasticsearch.yml
-----------------------------------------------------------------------------------
cluster.name: ELK-Cluster #ELK的集群名称，名称相同即属于是同一个集群
node.name: elk-node2 #本机在集群内的节点名称
path.data: /ELK/elasticsearch/data #数据存放目录
path.logs: /ELK/elasticsearch/data/log #日志保存目录
bootstrap.memory_lock: true #服务启动的时候锁定足够的内存，防止数据写入swap
network.host: 172.18.12.166 #监听的IP地址
http.port: 9200 #服务监听的端口
discovery.seed_hosts: ["172.18.12.172", "172.18.12.173"] #单播配置一台即可
cluster.initial_master_nodes: ["elk-node1", "elk-node2"]
-----------------------------------------------------------------------------------
[root@localhost config]# vim jvm.options
-----------------------------------------------------------------------------------
-Xms2g
-Xmx2g
-----------------------------------------------------------------------------------
[root@localhost config]# mkdir -p /ELK/elasticsearch/{data,data/log}
[root@localhost elasticsearch]# groupadd elk
[root@localhost elasticsearch]# useradd -g elk elk
[root@localhost elasticsearch]# passwd elk
[root@localhost elasticsearch]# chown -R elk:elk /ELK/elasticsearch/ -R
[root@localhost bin]# su elk
[elk@localhost elasticsearch]# cd bin/
[elk@localhost bin]# ./elasticsearch
[elk@localhost bin]$ yum -y install net-tools
[elk@localhost bin]# netstat -nltup |grep javatcp6 0 0 172.18.12.164:9200 :::* LISTEN 9971/java
tcp6 0 0 172.18.12.164:9300 :::* LISTEN 9971/java


Elasticsearch-head插件

一、解压并安装head插件&npm

[root@localhost tmp]# tar -zxvf elasticsearch-head.tar.gz
[root@localhost tmp]# mv elasticsearch-head /ELK/elasticsearch-head
[root@localhost tmp]# xz -d node-v12.14.0-linux-x64.tar.xz
[root@localhost tmp]# tar -xvf node-v12.14.0-linux-x64.tar
[root@localhost ELK]# ln -s /ELK/nodejs/bin/node /usr/bin/node
[root@localhost tmp]# mv node-v12.14.0-linux-x64 /ELK/nodejs
[root@localhost ELK]# ln -s /ELK/nodejs/bin/npm /usr/bin/npm //建立软连接
[root@localhost elasticsearch-head]# npm install --registry=https://registry.npm.taobao.org //使用淘宝源
[root@localhost elasticsearch-head]# npm run start
[root@localhost elasticsearch-head]# npm run start & //后台启动


二、访问http://IP___:9100

三、分别修改所有elasticsearch服务配置文件，开启跨域访问支持，然后重启elasticsearch服务

[root@localhost elasticsearch]# vim /ELK/elasticsearch/config/elasticsearch.yml


配置如下：

http.cors.enabled: true
http.cors.allow-origin: "*"


[root@localhost bin]# ps -ef |grep elasticelk 9971 1 0 13:22 pts/0 00:00:45 /tmp/jdk1.8.0_221/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.JlwA3m72 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFil[root@localhost bin]# kill -9 9971
[root@localhost bin]# su elk
[elk@localhost bin]$ ./elasticsearch &


四、浏览器访问9100端口，将连接地址修改为elasticsearch地址

五、提交测试数据

Master和Slave的区别：

Master的职责：
统计各node节点状态信息、集群状态信息统计、索引的创建和删除、索引分配的管理、关闭node节点等
Savle的职责：
同步数据、等待机会成为Master

Logstash部署

一、测试标准输入输出

[root@localhost tmp]$ mv logstash-7.5.1 /ELK/logstash
[root@localhost ELK]# cd logstash/bin/
[root@elk-node1 logstash]# vim /ELK/logstash/config/logstash.yml


配置logstash.yml

http.host: "192.168.30.129"
http.port: 9600


测试标准输入输出

[root@localhost bin]# ./logstash -e 'input { stdin {} } output { stdout { codec => rubydebug} }'


输入:hello word

{"@version" => "1","@timestamp" => 2019-12-19T07:33:46.725Z,"message" => "hello word","host" => "localhost"
}


二、测试输出到文件

[root@localhost bin]# ./logstash -e 'input { stdin{} } output { file { path => "/tmp/log-%{+YYYY.MM.dd}messages.gz"}}'


输入：

hello word

[2019-12-19T15:44:02,238][INFO ][logstash.outputs.file ][main] Opening file {:path=>"/tmp/log-2019.12.19messages.gz"}
[2019-12-19T15:44:20,886][INFO ][logstash.outputs.file ][main] Closing file /tmp/log-2019.12.19messages.gz


查看文件内容:

[root@localhost bin]# tail /tmp/log-2019.12.19messages.gz {"message":"hello world","@version":"1","@timestamp":"2019-12-19T07:44:01.261Z","host":"localhost"}


三、测试输出到elasticsearch

[root@linux-elk1 ~]# ./logstash -e 'input { stdin{} } output { elasticsearch {hosts => ["172.18.12.164:9200"] index => "mytest-%{+YYYY.MM.dd}" }}'


四、elasticsearch服务器验证收到数据

[root@localhost /]# ll /ELK/elasticsearch/data/nodes/0/indices/总用量 0
drwxrwxr-x. 8 elk elk 65 12月 19 15:51 bOpOoG7xRQW7OVV06ufvAw
drwxrwxr-x. 8 elk elk 65 12月 19 14:51 mPiADPveTbyGdvj-i9W7Uw


kibana部署

一、解压并配置 kibana.yml

[root@localhost tmp]# tar -zxvf kibana-7.5.1-linux-x86_64.tar.gz
[root@localhost tmp]# mv kibana-7.5.1-linux-x86_64 /ELK/kibana
[root@localhost tmp]# cd /ELK/kibana/config/
[root@localhost config]# vim kibana.yml


配置如下

server.port: 5601 #监听端口
server.host: "172.18.12.164" #监听地址
elasticsearch.hosts: ["http://172.18.12.164:9200","http://172.18.12.166:9200"] #elasticsearch服务器地址
i18n.locale: "zh-CN" #修改为中文


Elastic Stack实战

一、收集yum日志

[root@elk-node1 logstash]# mkdir conf.d
[root@elk-node1 logstash]# touch conf.d/yum-log.conf
[root@elk-node1 logstash]# vim conf.d/yum-log.conf


编辑新增配置文件：

input {file {path => "/var/log/yum.log" type => "systemlog" start_position => "beginning" stat_interval => "2" }
}output {elasticsearch {hosts => ["172.18.12.172:9200"] index => "logstash-%{type}-%{+YYYY.MM.dd}" }
}


检测配置文件语法是否正确：

[root@elk-node1 logstash]# /ELK/logstash/bin/logstash -f /ELK/logstash/conf.d/yum-log.conf -t


Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /ELK/logstash/logs which is now configured via log4j2.properties
[2019-12-20T16:53:29,595][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/ELK/logstash/data/queue"}
[2019-12-20T16:53:29,615][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/ELK/logstash/data/dead_letter_queue"}
[2019-12-20T16:53:30,038][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-12-20T16:53:31,328][INFO ][org.reflections.Reflections] Reflections took 67 ms to scan 1 urls, producing 20 keys and 40 values
Configuration OK
[2019-12-20T16:53:31,979][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash


二、重启logstash服务

[root@elk-node1 logstash]# /ELK/logstash/bin/logstash -f /ELK/logstash/conf.d/yum-log.conf &


三、Kibana可视化界面配置