Timezone offset parse issue

Elastic Stack Logstash
1

When i try to parse the following line

2018-04-17T10:04:15.693+0200 INFO [dmt_scheduler_Worker-4] [SampleClass:60] testMethod executed

I found the following error in the log file

{:timestamp=>"2018-04-17T10:04:15.693000+0200", :message=>"Failed parsing date from field", :field=>"timestamp", :value=>"2018-04-17 10:04:15.693", :exception=>"Invalid format: "2018-04-17 10:04:15.693" is malformed at " 10:04:15.693"", :config_parsers=>"yyyy-MM-dd'T'HH:mm:ss.SSSZZ", :config_locale=>"default=en_US", :level=>:warn}

My filter configuration is as follows

if "timeformat_java" in [tags] {
grok {
match => {
"message" => "^%{TIMESTAMP_ISO8601:timestamp}"
}
}
grok {
match => {
"message" => "%{LOGLEVEL:loglevel}"
}
}
grok {
match => {
"message" => "%{JAVAEX:exception}"
}
}

date {
  # 2015-06-26 09:45:14,439
  match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
}
kv { prefix => "param_" }

}

Please help me resolve this issue.

My expected output for timestamp field is 2018-04-17T10:04:15.693+0200, but it is showing as 2018-04-17T08:04:15.693Z

2

I can't see what the problem with the date filter is. Your grok filters are not ideal, they should all be in one grok pattern.
When I try this...

input {
  generator {
    message => '2018-04-17T10:04:15.693+0200'
    count => 1
  }
}

filter {
  date {
    match => [ "message", "yyyy-MM-dd'T'HH:mm:ss.SSSZZ"]
    target => "timestamp"
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

I correctly get...

{
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
      "sequence" => 0,
    "@timestamp" => 2018-04-17T08:42:49.551Z,
       "message" => "2018-04-17T10:04:15.693+0200",
     "timestamp" => 2018-04-17T08:04:15.693Z
}

This is correct because Logstash and Elasticsearch requires dates in UTC - they are actually UNIX epoch floating point numbers under the hood.

Note that:

  • "message" => "2018-04-17T10:04:15.693+0200", shows the message field in its UTF-8 String representation for humans. Under the hood it is "message" => 32 30 31 38 2D 30 34 2D 31 37 54 31 30 3A 30 34 3A 31 35 2E 36 39 33 2B 30 32 30 30, but that is not useful to humans.
  • "timestamp" => 2018-04-17T08:04:15.693Z shows the timestamp field representation for humans. Under the hood it is "timestamp" => 1523952255.693 but again thats not useful to humans. Instead it shows the human representation in terms of UTC.

The strings "2018-04-17T10:04:15.693+0200" and "2018-04-17T08:04:15.693Z" parse to the same Time instance -> 1523952255.693
The first string is formatted for display to a human in a timezone 2 hours ahead of UTC.

1 Like
3

Thank you very much for your reply.
I got your point.

Is it possible to get the following output?

{
"@version" => "1",
"host" => "Elastics-MacBook-Pro.local",
"sequence" => 0,
"@timestamp" => 2018-04-17T08:42:49.551Z,
"message" => "2018-04-17T10:04:15.693+0200",
"timestamp" => 2018-04-17T10:04:15.693+0200
}

4

No. When using the rubydebug codec, the human format of a timestamp data structure is always in UTC.

Where do you want the UTC+2 formatted string to show up? Kibana?

5

Your "No" answers my question, thank you very much for your time.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.