Logstash date filter not replacing @timestamp field

Elastic Stack Logstash
1

Hi all,
Please assist, as I'm already becoming desperate.
I'm unable to replace the @timestamp field with the actual log time.
I've tried all kinds of help from the internet but no luck.
This is my logstash filter date config:
date {
match => [ "logtime", "ISO8601"]
timezone => "UTC"
target => "@timestamp"
add_tag => [ "timestamp_match" ]
}
This is an example of grok filter:
grok {
match => [
"message", "%{TIMESTAMP_ISO8601:logtime} [PID %{NUMBER:pid}]\t[%{WORD:levelname}]\t[%{GREEDYDATA:content}"
]
}

This is the debug:
[2017-02-17T18:00:29,327][DEBUG][logstash.pipeline ] output received {"event"=>{"offset"=>4760815, "input_type"=>"log", "pid"=>"17206", "source"=>"/etc/qa/log/qamt1-all.log", "message"=>"2017-02-17 18:00:21,341 [PID 17206]\t[INFO]\t[test]\tmiddleware.py:11\ttenant.middleware: handling request for tenant test", "type"=>"django", "content"=>"handling request for tenant test", "tags"=>["beats_input_codec_plain_applied", "qa_server", "dashboards"], "@timestamp"=>2017-02-17T18:00:24.311Z, "filename"=>"middleware.py", "lineno"=>"11", "@version"=>"1", "beat"=>{"hostname"=>"test.net", "name"=>"test.net", "version"=>"5.1.2"}, "host"=>"test.net", "name"=>"tenant.middleware", "levelname"=>"INFO", "logtime"=>"2017-02-17 18:00:21,341", "tenant"=>"test"}}

in the kibana I see both @timestamp and logtime fields, which are not identical. Differences are in milliseconds, but it is very hard to read with millions of logs per minute.

I appreciate all the help I can get!!
Thank you!

2

Try escaping the characters in grok pattern. Change
[ TO \[
\t TO \\t

Try this for debugging. http://grokdebug.herokuapp.com/

3

Since filters are processed in order the date filter that parses the logtime field needs to come after the grok filter that creates the logtime field.

4

Thanks Magnus,
That was the problem.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.