Replace @timestamp with SYSLOGTIMESTAMP

[vuvu]

hi! I want to replace the @timestamp from the Dashboard with the timestamp from the logs that I get from some servers. The thing is that these logs have the SYSLOGTIMESTAMP format and just by using the date filter, it did not work.

This is my filter configuration:

filter {
  grok {
    match => { %{SYSLOGTIMESTAMP:syslog_timestamp} }
  }

  date {
    match => [ "syslog_timestamp",  "MMM  d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601" ]
    target => "@timestamp"
  }
}

This is how the logs look in the dashboard:

As you can see, @timestamp is different that the timestamp from event.original. I want @timestamp to be the same as the one from the log.

I saw some answers to similar questions about using the mutate filter to copy the timestamp, but I do not know how to do that given that SYSLOGTIMESTAMP is a different format than that of @timestamp. Please help!

[leandrojmp (Leandro Pereira)]

Hello,

Please share a plain text sample of your log so people can try to replicate your issue.

Also, your grok filter seems to be wrong, you didn't specified which field it should use, check the documentation.

It would need to be something like this:

grok {
    match => {
        "message" =>  "%{SYSLOGTIMESTAMP:syslog_timestamp}" 
    }
}

[vuvu]

Logs for testing:

Feb  7 12:35:56 hostnamexxxxxxxxxx systemd: Created slice User Slice of root.
Feb  7 12:30:01 hostnamexxxxxxxxxx systemd: Started Session 9520 of user root.
Feb  7 12:12:59 hostnamexxxxxxxxxx systemd-logind: Removed session 3893.

[vuvu]

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}" }
  }

  date {
    match => [ "syslog_timestamp",  "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
    timezone => "Europe/Amsterdam"
    target => "@timestamp"
  }
}

I set the timezone the same as it is on the server, but I still have one hour difference between the logs.

[leandrojmp (Leandro Pereira)]

@vuvu

I set the timezone the same as it is on the server, but I still have one hour difference between the logs.

Were you able to populate the @timestamp field?

In which timezone are your logs being generated?

For example, this time in your log Feb 7 12:35:56 was generated in which timezone?

[vuvu]

I was not able to solve this yet.

The logs on the server are created on the "Europe/Amsterdam" timezone, which I also set recently in the date filter, but to no avail. This is how the code looks now:

filter {
  grok {
    match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp}" }
  }

  date {
    match => [ "syslog_timestamp",  "MMM  d HH:mm:ss", "MMM dd HH:mm:ss"]
    timezone => "Europe/Amsterdam"
    locale => "en"
    target => "@timestamp"
  }
}

This is an example of the variables output to the console for one of the logs:

 "syslog_timestamp" => "Feb  7 15:37:45",
 "@timestamp" => 2024-02-07T14:37:45.000Z

This is very weird to me, because @timestamp on the Dashboard is not 14:37:45 (like the console output), but it is 16:37:45.

[leandrojmp (Leandro Pereira)]

So, the date filter is working correctly.

It is converting the Amsterdam time (UTC+1) to UTC.

 "syslog_timestamp" => "Feb  7 15:37:45",
 "@timestamp" => 2024-02-07T14:37:45.000Z

This is correct.

But when this logs is rendered on Kibana, all date fields will be converted back to the timezone configured in Kibana.

Did you change the timezone settings in Kibana? What is the timezone of the operating system where this browser is running? From what you shared it seems that it is in a timezone of UTC+2, since it is adding 2 hours to the UTC time.

For example, what is the time in the operating system running this browser?

[vuvu]

ah, I get now how the date filter works. Thank you!
The timezone is indeed UTC+2, but I do not have access to change the settings in Kibana.

In this case, shoud I try to substract one hour from the log's timestamp using a ruby script and then use the date filter on it? I think this way it would display the correct timestamp when converting to UTC+2 in Kibana.

[leandrojmp (Leandro Pereira)]

@vuvu

The timezone is indeed UTC+2

If you are on UTC+2 then Kibana is correctly converting the time from UTC to your local timezone, I don't think anything needs to be changed here.

For example, you have an event that was logged using amsterdam timezone and you correctly informed the date filter about it so the event could be converted to UTC, but you are also using Kibana on a third different timezone, which is UTC+2, you will have this:

• Original timestamp: Feb 7 15:37:45
• UTC timestamp: Feb 7 14:37:45
• Kibana timestamp: Feb 7 16:37:45

Those are all the same time, so not sure what you want to change here.

@vuvu

In this case, shoud I try to substract one hour from the log's timestamp using a ruby script and then use the date filter on it? I think this way it would display the correct timestamp when converting to UTC+2 in Kibana.

All your times are already correct as mentioned before.

15:37 (UTC +1) = 14:37 (UTC) = 16:37 (UTC+2).

If you subtract one hour of original log and still use the timezone as Amsterdam, you are basically telling that you event happened one hour earlier.

I strongly recommend against changing the original log by subtracting or adding times as this can lead to inconsistency to other log sources and also some confusion to people using Kibana.

If you want Kibana to show the field syslog_timestamp also converted to UTC, you will need to map this field as a date field.