好吧,经过四处寻找,我无法找到问题的解决方案,因为它"应该"有效,但显然没有.我正在使用Ubuntu 14.04 LTS机器Logstash 1.4.2-1-2-2c0f5a1,我收到的消息如下:
2014-08-05 10:21:13,618 [17] INFO Class.Type - This is a log message from the class: BTW, I am also multiline
在输入配置中,我有一个multiline
编解码器,并且事件被正确解析.我还将事件文本分成几个部分,以便于阅读.
最后,我在Kibana中看到了类似下面的内容(JSON视图):
{ "_index": "logstash-2014.08.06", "_type": "customType", "_id": "PRtj-EiUTZK3HWAm5RiMwA", "_score": null, "_source": { "@timestamp": "2014-08-06T08:51:21.160Z", "@version": "1", "tags": [ "multiline" ], "type": "utg-su", "host": "ubuntu-14", "path": "/mnt/folder/thisIsTheLogFile.log", "logTimestamp": "2014-08-05;10:21:13.618", "logThreadId": "17", "logLevel": "INFO", "logMessage": "Class.Type - This is a log message from the class:\r\n BTW, I am also multiline\r" }, "sort": [ "21", 1407315081160 ] }
您可能已经注意到我放了一个";" 在时间戳中.原因是我希望能够使用时间戳字符串对日志进行排序,显然logstash并不是那么好(例如:http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/ multi-fields.html).
我没有成功尝试过date
多种方式使用过滤器,显然它没有用.
date { locale => "en" match => ["logTimestamp", "YYYY-MM-dd;HH:mm:ss.SSS", "ISO8601"] timezone => "Europe/Vienna" target => "@timestamp" add_field => { "debug" => "timestampMatched"} }
因为我读到Joda库可能会遇到问题,如果字符串不是严格符合ISO 8601标准(非常挑剔并期待T,请参阅https://logstash.jira.com/browse/LOGSTASH-180),我也尝试过用于mutate
将字符串转换为类似2014-08-05T10:21:13.618
然后使用"YYYY-MM-dd'T'HH:mm:ss.SSS"
.这也行不通.
我不想在时间上手动输入+02:00,因为这会给夏令时带来问题.
在任何一种情况下,事件进入elasticsearch,但date
确实显然没有,因为@timestamp
和logTimestamp
是不同的,没有debug
加场.
知道如何让logTime字符串正确排序吗?我专注于将它们转换为适当的时间戳,但任何其他解决方案也将受到欢迎.
如下所示:
当排序过@timestamp
,elasticsearch可以做正确,但由于这是不是"真正的"日志时间戳,而是当读取的logstash事件,我需要(明显),以便能够在同时进行排序logTimestamp
.这就是输出.显然没那么有用:
欢迎任何帮助!如果我忘记了一些可能有用的信息,请告诉我.
更新:
这是最终工作的过滤器配置文件:
# Filters messages like this: # 2014-08-05 10:21:13,618 [17] INFO Class.Type - This is a log message from the class: # BTW, I am also multiline # Take only type- events (type-componentA, type-componentB, etc) filter { # You cannot write an "if" outside of the filter! if "type-" in [type] { grok { # Parse timestamp data. We need the "(?m)" so that grok (Oniguruma internally) correctly parses multi-line events patterns_dir => "./patterns" match => [ "message", "(?m)%{TIMESTAMP_ISO8601:logTimestampString}[ ;]\[%{DATA:logThreadId}\][ ;]%{LOGLEVEL:logLevel}[ ;]*%{GREEDYDATA:logMessage}" ] } # The timestamp may have commas instead of dots. Convert so as to store everything in the same way mutate { gsub => [ # replace all commas with dots "logTimestampString", ",", "." ] } mutate { gsub => [ # make the logTimestamp sortable. With a space, it is not! This does not work that well, in the end # but somehow apparently makes things easier for the date filter "logTimestampString", " ", ";" ] } date { locale => "en" match => ["logTimestampString", "YYYY-MM-dd;HH:mm:ss.SSS"] timezone => "Europe/Vienna" target => "logTimestamp" } } } filter { if "type-" in [type] { # Remove already-parsed data mutate { remove_field => [ "message" ] } } }
Ben Lim.. 19
我测试了你的date
过滤器.它适用于我!
这是我的配置
input { stdin{} } filter { date { locale => "en" match => ["message", "YYYY-MM-dd;HH:mm:ss.SSS"] timezone => "Europe/Vienna" target => "@timestamp" add_field => { "debug" => "timestampMatched"} } } output { stdout { codec => "rubydebug" } }
我用这个输入:
2014-08-01;11:00:22.123
输出是:
{ "message" => "2014-08-01;11:00:22.123", "@version" => "1", "@timestamp" => "2014-08-01T09:00:22.123Z", "host" => "ABCDE", "debug" => "timestampMatched" }
因此,请确保您logTimestamp
的值正确.这可能是其他问题.或者您可以提供日志事件和logstash配置以进行更多讨论.谢谢.
我测试了你的date
过滤器.它适用于我!
这是我的配置
input { stdin{} } filter { date { locale => "en" match => ["message", "YYYY-MM-dd;HH:mm:ss.SSS"] timezone => "Europe/Vienna" target => "@timestamp" add_field => { "debug" => "timestampMatched"} } } output { stdout { codec => "rubydebug" } }
我用这个输入:
2014-08-01;11:00:22.123
输出是:
{ "message" => "2014-08-01;11:00:22.123", "@version" => "1", "@timestamp" => "2014-08-01T09:00:22.123Z", "host" => "ABCDE", "debug" => "timestampMatched" }
因此,请确保您logTimestamp
的值正确.这可能是其他问题.或者您可以提供日志事件和logstash配置以进行更多讨论.谢谢.