Overwriting json @timestamp


(Frank Solli) #1

Hello, I am sending json logs to logstash and I want to overwrite the @timestamp field.

To do this I do:

json.keys_under_root: true
json.overwrite_keys: true

The log entries goes to logstash, but in Kibana the @timestamp field is not overwritten and shows the error:

@timestamp not overwritten (parse error on 2017-06-02T21:40:59+0000)


(Andrew Kroh) #2

It looks like Go's RFC3389 time parser is failing on that timestamp. This recreates the issue. The full error is:

parsing time "2017-06-02T21:40:59+0000" as "2006-01-02T15:04:05Z07:00": cannot parse "+0000" as "Z07:00"

Maybe the time parser should be a bit more robust and try a few more common formats.

For now you will need to use Logstash to handle this.


(Frank Solli) #3

Thank you!

Do you have any suggestions on how I can fix this with logstash?


(Andrew Kroh) #4

One way would be to do the JSON decoding in Logstash. Then apply a date filter.

filter {
  json {
    source => "message"
  }
  date {
     # Add config here for parsing the date.
  }
}

https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html#plugins-filters-json-source
https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-target


(Frank Solli) #5

I added this now, but I still get the same error message:

input {
beats {
port => 5445
codec => "json"
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}

filter {
json {
source => "message"
}
date {
match => [ "timestamp", "ISO8601" ]
}
}


(Joseph Johney) #6

Try using@timestamp instead


(Frank Solli) #7

This made no difference.


(Andrew Kroh) #8

Remove that line.


(Frank Solli) #9

Hi, thanks for the suggestion, but this did not make any difference.


(Andrew Kroh) #10

Did you disable the JSON parsing on the Beats side?


(Frank Solli) #11

Yes, tried with and without.


(Frank Solli) #12

Can this be due to the timestamp beeing

2017-06-02T21:40:59+0000

And not

2017-06-02T21:40:59+00:00

Which is the correct ISO8601 format?


(system) #13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.