Logstash mapping logfile time to @timestamp

[Vajb12 (Vajreswari)]

Hi Team,

Im trying to map the logfile time with @timestamp in logstash. The logfile time is in IST. I would require some help in converting the logfile time to UTC and map it to @timestamp.
I used the below ruby code to change the log file time to UTC.

ruby {

code => "event.set( 'logts' , Time.now.utc)"
}

if [logts] {
date {
match => [ "logts", "ISO8601" ]
target => ["@timestamp"]
#target => ["logts"]

}
}

since i have used Time.now.utc, it converts all the logtime (logts) to current utc time. Which is incorrect. Can you please help in solving the above issue. Any help on this is much appreciated. Thanks in Advance.

[leandrojmp (Leandro Pereira)]

Please, share an example of your log with the time field.

You don't need to use ruby to create a field with the UTC time, if your time field in your log is in another timezone you can use the timezone option of the date filter.

[Vajb12 (Vajreswari)]

@leandrojmp Thanks a lot for your response. Below is an example of the my log with time field

"message":" [ERROR] **2021-07-13T19:43:44.552** [http-nio2-8080-exec-1092] amrwebdeployment-5cb5b4f68b-2zh4z - [i.i.i.a.f.AuthorizationFilter] :: The classmap is not loaded for the ITDRuntimeException with error code: AMR-RUNTIME-MS-9021 "

The grok filter which Im using to filter the above data is

grok { match => { "mesg" => [ "^\s?[%{DATA:loglevel}] %{TIMESTAMP_ISO8601:logts} [%{DATA:threadname}] %{DATA:podname} %{DATA:filler1} [%{DATA:classname}] %{GREEDYDATA:fullmesg}"] } }

The logfile time is in IST timing and I want to set/map logts time to the @timestamp

[leandrojmp (Leandro Pereira)]

You can use the timezone option in the date filter.

Remove your ruby filter as it is just override the logts field with the current utc time and try this filter.

date {
  match => ["logts", "ISO8601"]
  timezone => "your/timezone"
}

IST is ambiguous, it could mean India Standard Time, Irish Standard Time or Israel Standar Time, you need the name of the timezone you are, which would be Asia/Kolkata, Europe/Dublin or Asia/Jerusalem for each one of those cases.

In the case IST means India Standard Time you would use:

date {
  match => ["logts", "ISO8601"]
  timezone => "Asia/Kolkata"
}

[Vajb12 (Vajreswari)]

@leandrojmp - The log time I meant IST was Indian standard time. Before using the ruby code, I first used the timezone option only. I tried two time zone options. when I set the timezone => "Asia/Kolkata" in the date filter, the resulting date (log date) in kibana seems to show a future date. When I set the timezone => "UTC " in the Date filter, same behaviour is observed. Hence I went with the Ruby filter . Please suggest.

[leandrojmp (Leandro Pereira)]

There is not much to suggest.

If your date is in the Asia/Kolkata timezone, then you need to tell that to the date filter, so it will convert correctly to the UTC time as elasticsearch only stores date in UTC.

What is the timezone of the machine running Kibana? Kibana per default will convert the UTC times to the browser timezone, which will use the time zone of the operating system.

You can change this behavior in Kibana going into Stack Management > Kibana Advanced Settings > Timezone for date formatting, there you can choose which timezone Kibana should use.