1、Logstash是一个数据收集引擎&＃xff0c;相当于是ETL工具。截图来源慕课&＃xff0c;尊重版本从你我做起。
Logstash分为三个阶段&＃xff0c;第一个阶段Input是数据采集、第二个阶段是Filter数据解析和转换&＃xff0c;第三个阶段是Output数据输出。
2、Logstash中的Pipeline概念。
1)、Pipeline是指Input-filter-output的三个阶段处理流程。
2)、队列管理。
3)、插件生命周期管理。3、Logstash中的Logstash Event概念。
1)、内部流转的数据表现形式&＃xff0c;原始数据从Input进入之后&＃xff0c;在内部流转的时候不是原始的数据&＃xff0c;而是Logstash Event数据。Logstash Event就是一个Java Object&＃xff0c;对外暴漏去修改或者获取内部字段的api
2)、原始数据在input被转换为Event&＃xff0c;在output event被转换为目标格式数据。
3)、在配置文件中可以对Event中的属性进行增删改查。
关于数据流转&＃xff0c;数据由Input进入之后&＃xff0c;从Output输出。Codec(Code、Decode)将原始数据Data转换为Logstash Event&＃xff0c;当数据输出的时候&＃xff0c;Codec将Logstash Event转换为目标数据源需要的类型Data。
4、Logstash的安装&＃xff0c;将包上传到服务器进行解压缩即可&＃xff0c;如下所示&＃xff1a;
1 tar -zxvf logstash-6.7.1.tar.gz -C /usr/local/soft/
Logstash的简单案例&＃xff0c;如下所示&＃xff1a;
1 # 输入&＃xff0c;stdin是标准输入&＃xff0c;按照每一行切分数据2 input {3 stdin {4 codec &＃61;> line5 }6 }7 8 # 过滤为空9 filter {}
10
11 # 输出&＃xff0c;stdout标准输出&＃xff0c;将输出转换为json格式
12 output {
13 stdout {
14 codec &＃61;> json
15 }
16 }
以每行进行切分数据&＃xff0c;不过滤&＃xff0c;然后输出为json格式的输入(Codec- Input Decoding)转换流程&＃xff0c;如下所示&＃xff1a;
当输入之后&＃xff0c;要进行输出(Codec- Output Encoding)&＃xff0c;如下所示&＃xff1a;
演示效果&＃xff0c;如下所示&＃xff1a;
1 [elsearch&＃64;k8s-master logstash-6.7.1]$ echo "foo2 > bar3 > " | bin/logstash -f config/line-to-json.conf4 Sending Logstash logs to /usr/local/soft/logstash-6.7.1/logs which is now configured via log4j2.properties5 [2021-01-30T19:11:13,330][INFO ][logstash.setting.writabledirectory] Creating directory {:setting&＃61;>"path.queue", :path&＃61;>"/usr/local/soft/logstash-6.7.1/data/queue"}6 [2021-01-30T19:11:13,357][INFO ][logstash.setting.writabledirectory] Creating directory {:setting&＃61;>"path.dead_letter_queue", :path&＃61;>"/usr/local/soft/logstash-6.7.1/data/dead_letter_queue"}7 [2021-01-30T19:11:14,189][WARN ][logstash.config.source.multilocal] Ignoring the &＃39;pipelines.yml&＃39; file because modules or command line options are specified8 [2021-01-30T19:11:14,258][INFO ][logstash.runner ] Starting Logstash {"logstash.version"&＃61;>"6.7.1"}9 [2021-01-30T19:11:14,396][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid&＃61;>"e449e4d8-b26e-49fa-b12d-3d452b50aac4", :path&＃61;>"/usr/local/soft/logstash-6.7.1/data/uuid"}
10 [2021-01-30T19:11:24,816][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id&＃61;>"main", "pipeline.workers"&＃61;>2, "pipeline.batch.size"&＃61;>125, "pipeline.batch.delay"&＃61;>50}
11 [2021-01-30T19:11:24,943][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id&＃61;>"main", :thread&＃61;>"#
12 [2021-01-30T19:11:25,080][INFO ][logstash.agent ] Pipelines running {:count&＃61;>1, :running_pipelines&＃61;>[:main], :non_running_pipelines&＃61;>[]}
13 {"&＃64;version":"1","host":"k8s-master","message":"foo","&＃64;timestamp":"2021-01-30T11:11:25.063Z"}{"&＃64;version":"1","host":"k8s-master","message":"bar","&＃64;timestamp":"2021-01-30T11:11:25.102Z"}{"&＃64;version":"1","host":"k8s-master","message":"","&＃64;timestamp":"2021-01-30T11:11:25.102Z"}[2021-01-30T19:11:25,394][INFO ][logstash.pipeline ] Pipeline has terminated {:pipeline_id&＃61;>"main", :thread&＃61;>"#
14 [2021-01-30T19:11:25,516][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port&＃61;>9600}
15 [elsearch&＃64;k8s-master logstash-6.7.1]$
京公网安备 11010802041100号