Need Help Creating Logstash Filter to use timestamp in log message in Elastic/Kibana

Elastic Stack Logstash
1

Hello,

Okay, I'm probably missing something pretty obvious.

I have a log message like the following:

INFO 2016-07-21 13:17:48,139 [http-bio-8080-exec-5] com.vendor.recserver.controller.RestController - site:company; abtest:none; pagetemplate:PT_RelatedRec: Total Time = 1ms. widget:RecentlyViewedProduct time:0ms scanned:0 timebox:none fallback:0 of 5 widget:RelatedRec time:0ms scanned:4 timebox:none fallback:0 of 4 context-url:http://www.company.com/eu/p/347340

Trying to create the fields so that elastic and kibana can do their magic, but I'm not getting far

Here's my logstash configuration

filter {
date {
match => ["logtime", "ISO8601" ]
}
grok {
match => { "message" => "%{LOGLEVEL:severity} %{TIMESTAMP_ISO8601:logtime} %{NOTSPACE:javathread} %{JAVACLASS:class} %{GREEDYDATA:therest}"
}
}
}

In elastic search and kibana, the date is not being used to index. Here's the message in json output from kibana

{
"_index": "logstash-2016.07.27",
"_type": "logs",
"_id": "AVYr5KBDX2kxhEGTGKoW",
"_score": 1,
"_source": {
"message": "INFO 2016-07-21 13:17:48,139 [http-bio-8080-exec-5] com.vendor.recserver.controller.RestController - site:company; abtest:none; pagetemplate:PT_RelatedRec: Total Time = 1ms. widget:RecentlyViewedProduct time:0ms scanned:0 timebox:none fallback:0 of 5 widget:RelatedRec time:0ms scanned:4 timebox:none fallback:0 of 4 context-url:http://www.company.com/eu/p/347340",
"@version": "1",
"@timestamp": "2016-07-27T10:26:57.948Z",
"host": "dc-lt-mbp15",
"severity": "INFO",
"logtime": "2016-07-21 13:17:48,139",
"javathread": "[http-bio-8080-exec-5]",
"class": "com.vendor.recserver.controller.RestController",
"therest": "- site:company; abtest:none; pagetemplate:PT_RelatedRec: Total Time = 1ms. widget:RecentlyViewedProduct time:0ms scanned:0 timebox:none fallback:0 of 5 widget:RelatedRec time:0ms scanned:4 timebox:none fallback:0 of 4 context-url:http://www.company.com/eu/p/347340"
},
"fields": {
"@timestamp": [
1469615217948
]
}
}

2

Filters are processed in the order they're listed. Your date filter uses the logtime field that's extracted by the grok filter so you need the grok filter to go first.

3

Hello Magnus,

Thank you for your reply. Yes, of course that worked. When I tried that before I thought it was not working because I forgot to look back using kibana! Once I realized that, I found that previous experiment had worked and I just had not looked for the properly time stamped message.

Thank you.

Cheers, Dario