Logstash Syslog @timestamp incorrect

Elastic Stack Logstash
1

Hi all,

I am running into a bit of an issue with Logstash 6.6.0 and it processing Syslog messages. For some reason it is setting the @timestamp field to America/New_York time thinking that it is UTC time (Kibana is also displaying it as if it thinks the field is UTC). I have been messing around with the issue all day, but I have been unable to find a solution to my problem.

Below is my logstash.yml

input {
  syslog {
    port => 5001
  }
}
filter {
  mutate {
    add_field => [ "received_at", "%{@timestamp}" ]
  date {
    timezone => "America/New_York"
    match => ["received_at", "ISO8601"]
    target => ["received_at"]
  }
}
output {
  stdout { codec => rubydebug }
}

However, I am still getting the incorrect output from this.

Below is an example of a rubydebug output.

{
      "severity" => 6,
"facility_label" => "local5",
       "message" => "10.8.5.85 - - [31/Jan/2019:20:57:32 -0500] \"GET /static/js/vendor/jquery.min.js HTTP/1.1\" 304 334 1157 258 \"https://website/admin\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0\" \"website\"\n",
     "logsource" => "website",
"severity_label" => "Informational",
     "timestamp" => "Jan 31 20:57:32",
      "priority" => 174,
   "received_at" => 2019-01-31T20:57:32.000Z,
          "host" => "website",
       "program" => "httpd_admin_access",
    "@timestamp" => 2019-01-31T20:57:32.000Z,
      "facility" => 21,
      "@version" => "1"
}

As you can see the timestamp is set to 2019-01-31T20:57:32.000Z, however this is actually the America/New_York time, in UTC time it should be 2019-02-01T01:57:32.000Z

2

The trailing Z means that timestamp is in Zulu (UTC) and nothing you put in the timezone option is going to change that.

3

Is there anything that can be done to fix this though? Because it is definitely not UTC time. Is this something that is wrong on the remote server end with rsyslog, or is this still an issue with Logstash?

4

@timestamp is always in UTC. Perhaps you should parse timestamp instead?

5

Perhaps there maybe some miscommunication with the issue at hand. I know that @timestamp always uses UTC as its timezone. The issue is that it is marking the @timestamp as UTC, but @timestamp is given the value of America/New_York timezone and not given the value of UTC timezone as it should be.

6

Specify the timezone option to the syslog input?

7

I was able to finally get this to set the correct time by using a very round about way.

mutate {
    add_field => [ "received_at", "%{@timestamp}" ]
}
grok {
    match => {  "received_at" => "%{TIMESTAMP_ISO8601:timestampNoZ}Z" }
}
date {
    timezone => "America/New_York"
    match => ["timestampNoZ", "ISO8601"]
}
mutate {
    remove_field => [ "received_at", "timestampNoZ" ]
}

While I am sure that this can be cleaned up, it does work in getting the timestamp to be adjusted to the correct value.

8

Did you try setting the timezone option on the syslog input? Did that not work?

9

I decided not to use the timezone input on the syslog input, because there are multiple hosts going through this pipeline, and only one of the hosts is sending the incorrect timestamp info. The code above is nested in a filter which only changes the affected host.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.