Copying @timestamp in custom field not working

Elastic Stack Logstash
1

Hi,
I am trying to copy the timestamp field in logs to some custom field 'eventLogTime' on Kibana. This I want to use for sorting on basis of event time.

I am using below configuration in logstash indexer

grok {
                 match => { "message" => "%{TIMESTAMP_ISO8601:eventLogTime}" }
                 add_field => [ "eventLogTime", "%{eventLogTime}" ]
         }
        date {
                 match => [ "eventLogTime", "YYYY-MM-dd'T'HH:mm:ss", "YYYY-MM-dd HH:mm:ss","HH:mm:ss MMM dd yyyy","YYYY-MM-dd HH:mm:ss,SSS","yyyy.MM.dd G 'at' HH:mm:ss z","yyyyy.MMMMM.dd GGG hh:mm aaa","EEE, d MMM yyyy HH:mm:ss Z","yyyy-MM-dd'T'HH:mm:ss.SSSZ","yyyy-MM-dd'T'HH:mm:ss.SSSZ+0300", "YYYY/MM/dd HH:mm:ss","yyyy-MM-dd'T'HH:mm:ss.SSSSSZ+03:00"]
        target => "eventLogTime"

However I ma not getting expected result.

Log line is

 2015-11-04 14:22:04,598 username:'' INFO  [DepartureProgressLoaderJob: 41] - DepartureProgressLoaderJob loaded: 0 departures in 228ms

I am expecting 2015-11-04 14:22:04,598 under the field evenLogTime on Kibana. But the time shown under eventLogTime is November 4th 2015, 17:52:04.598 . Notice the time difference which is exactly equal to the difference in EET and IST.

Please let me know what is wrong with my filter.

thanks
Sunil Chaudhari.

2

add_field => [ "eventLogTime", "%{eventLogTime}" ]

This doesn't make sense. You're assigning a field to itself. Remove.

I am expecting 2015-11-04 14:22:04,598 under the field evenLogTime on Kibana. But the time shown under eventLogTime is November 4th 2015, 17:52:04.598 . Notice the time difference which is exactly equal to the difference in EET and IST.

The date filter converts timestamps to UTC for storage in Elasticsearch and Kibana then adjusts UTC timestamps to the browser's timezone. This is by design and is currently not configurable.

3

Thanks Magnus,

Is there anyway to copy timestamp (in the log) to customized field. And then I will use that new time field instead of @timestamp for time filter.

4

Sorry, not following. How is what you describe different from what you're already doing? You parse the message to extract an eventLogTime field, and parse that into an ISO8601 timestamp.

If you want your timestamp field as used in Kibana to be named anything but @timestamp you have to make sure you map that field as a timestamp. If you look at the existing index template I think it'll be obvious what you need to do.

5

Hi Magnu,
Thanks. Now its clear.

I didnt read your last comment carefully. "Kibana then adjusts UTC timestamps to the browser's timezone"
My mistake.

I am doing exactly what I want, but I was confused because I am accessing Kibana in IST zone. In production it will be accessed from EET. so No issue. :smile:

Thanks,
Sunil.