Docker搭建ELK日志日志分析系统

最近公司要求搭建ELK日志系统将日志维护起来&＃xff0c;网上看没有几个能直接跑起来的&＃xff0c;遇到了挺多卡&＃xff0c;这里简单分享下配置

版本号

这里使用Docker搭建&＃xff0c;简化操作配置&＃xff0c;不说废话直接上图

1. Filebeat

filebeat.yml:&＃xff08;定义filebeat配置文件&＃xff09;

filebeat.inputs:
- type: logenabled: truepaths:- /你项目的路径/*.logscan_frequency: 10s #查询的频率#下面4行&＃xff0c;意思是将正则匹配不到的行合并到上一行的行尾multiline.type: patternmultiline.pattern: &＃39;^\[[INFO|ERROR|WARN]&＃39;multiline.negate: truemultiline.match: after#tags: ["logapp"]fields:index: "dispatcher"#如果设置为true&＃xff0c;Filebeat从文件尾开始监控文件新增内容&＃xff0c;把新增的每一行文件作为一个事件依次发送而不是>从文件开始处重新发送所有内容tail_files: false
#&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61; Filebeat modules &＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;&＃61;
filebeat.config.modules:path: ${path.config}/modules.d/*.ymlreload.enabled: trueoutput.logstash:hosts: ["IP:5044"] #IP为logstash安装的服务器ipenabled: true


启动filebeat&＃xff1a;./filebeat -e -c filebeat.yml

如果想多次抓取需要删除filebeat的data目录&＃xff0c;里面记载了当前查找的索引位置

2. Logstash

logstash.yml&＃xff1a;&＃xff08;定义logstash配置文件&＃xff09;

http.host: "0.0.0.0"
xpack.monitoring.elasticsearch.hosts: [ "http://es01:9200" ]
xpack.monitoring.enabled: false#权限
# xpack.monitoring.elasticsearch.username: "elastic"
# xpack.monitoring.elasticsearch.password: "123123"path.config: /usr/share/logstash/config/conf.d/*.conf
path.logs: /usr/share/logstash/logs#不转义\n等数据
config.support_escapes: false

conf.d目录下新建logstash.conf:&＃xff08;定义过滤管道&＃xff09;

input {beats {port &＃61;> "5044"}
}filter { grok{ match &＃61;> { "message"&＃61;>"\[%{LOGLEVEL:Level}\] %{TIMESTAMP_ISO8601:Timestamp} %{DATA:PackageName}\)<%{DATA:Thread}>" }} grok{ match &＃61;> { "message"&＃61;>">\n(?>.*?)\n" }} grok{ match &＃61;> { "message"&＃61;>".*(?org\S&＃43;?Exception)" }} grok{ match &＃61;> { "message"&＃61;>".*CallNo&＃61;(?\w&＃43;)" }} grok{ match &＃61;> { "message"&＃61;>".*CallSheetID&＃61;(?\S&＃43;?)&" }} grok{ match &＃61;> { "message"&＃61;>".*CalledNo&＃61;(?\w&＃43;)" }} date {match &＃61;> [ "Timestamp", "yyyy-MM-dd HH:mm:ss,SSS" ]}mutate{replace &＃61;> ["Hostname","%{[agent][hostname]}"]replace &＃61;> ["FilePath","%{[log][file][path]}"]remove_field &＃61;> [&＃39;host&＃39;,&＃39;ecs&＃39;,&＃39;&＃64;version&＃39;,&＃39;Timestamp&＃39;,&＃39;log&＃39;,&＃39;agent&＃39;,&＃39;input&＃39;,&＃39;tags&＃39;,&＃39;message&＃39;] }
}output {stdout {codec &＃61;> rubydebug}elasticsearch {hosts &＃61;> [ "IP:9200" ]index &＃61;> "%{[fields][index]}"manage_template &＃61;> truetemplate&＃61;>"/usr/share/logstash/templates/dispatcher_template.json"template_name&＃61;>"dispatcher_template"template_overwrite&＃61;>true#权限
# user &＃61;> "elastic"
# password &＃61;> "123123"}
}


templates下新建dispatcher_template.json&＃xff1a;&＃xff08;定义logstash静态模版&＃xff09;

{"order": 10,"template": "dispatcher*","settings": {"index": {"refresh_interval": "60s","number_of_shards": "5","store": {"type": "fs"},"number_of_replicas": "0"}},"mappings": {"dispatcher":{"dynamic": "strict","properties": {"&＃64;timestamp": {"format":"yyyy-MM-dd HH:mm:ss,SSS||yyyy-MM-dd||epoch_millis","type": "date"}"Hostname": {"store": true,"type": "completion"},"CallSheetID": {"store": true,"type": "keyword"},"CallNo": {"store": true,"type": "keyword"},"CalledNo": {"store": true,"type": "keyword"},"PackageName": {"store": true,"type": "keyword"},"Thread": {"store": true,"type": "keyword"},"Exception": {"store": true,"type": "completion"},"Test": {"search_analyzer": "ik_smart","analyzer":"ik_max_word","store": true,"type": "text"}}}}
}


3. Es

conf下新建elasticsearch.yml&＃xff1a;&＃xff08;定义es配置文件&＃xff09;

---
## Default Elasticsearch configuration from Elasticsearch base image.
## https://github.com/elastic/elasticsearch/blob/master/distribution/docker/src/docker/config/elasticsearch.yml
#
cluster.name: "es-docker-cluster"
network.host: 0.0.0.0## X-Pack settings 开启权限
xpack.security.enabled: false
# xpack.security.transport.ssl.enabled: true
# xpack.license.self_generated.type: basic#跨域
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type


docker-compose.yml&＃xff1a;&＃xff08;编排容器-单体&＃xff09;

version: &＃39;3.7&＃39;
services:es01:image: elasticsearch:7.16.1container_name: es01volumes:- /你的地址/es/conf/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml- /你的地址/es/node01/data:/usr/share/elasticsearch/data- /你的地址/es/plugins:/usr/share/elasticsearch/pluginsports:- "9200:9200"environment:- discovery.type&＃61;single-node #单节点设置- bootstrap.memory_lock&＃61;true- "ES_JAVA_OPTS&＃61;-Xms512m -Xmx512m"ulimits:memlock:soft: -1hard: -1networks:- elastickibana:image: kibana:7.16.1container_name: kibana_clientvolumes:- /你的地址/kibana/kibana.yml:/usr/share/kibana/config/kibana.yml:rwports:- "5601:5601"networks:- elasticdepends_on:- logstash- es01logstash:image: logstash:7.16.1container_name: logstashcommand: logstash -f /usr/share/logstash/config/conf.d/logstash.confports:- "9600:9600"- "5044:5044"volumes:- /你的地址/logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml- /你的地址/logstash/config/conf.d:/usr/share/logstash/config/conf.d- /你的地址/logstash/config/templates:/usr/share/logstash/templatesnetworks:- elasticdepends_on:- es01networks:elastic:driver: bridge

然后启动docker-compose up