Date-filter - Issue with setting @timestamp with the value from my log

Elastic Stack Logstash
1

Hi Lads,
First of all i'm a newbie here.........

I was trying to format below log entry
2019-04-29 12:47:46.679.9882 - log message "blah blah blah"

i was testing it with below stdin

input {
   stdin { }
}
filter {
    grok {
       match => { "message" => "%{NOTSPACE}%{TIMESTAMP_ISO8601:logdate}%{GREEDYDATA:msg}" }
    }
    date {
        #match => ["logdate", "ISO8601"]
        match => ["logdate", "yy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    }
}
output {
    stdout { codec => rubydebug }
}

However, when I pass my log entry the output I get is as below

^X2019-04-29 12:47:46.679.9882 - log message "blah blah blah"
#Yup there is this weird first charactor in my log, so im escaping it with a "%{NOTSPACE}"
{
          "host" => "thanu00-elk7-01",
      "@version" => "1",
       "message" => "\u00182019-04-29 12:47:46.679.9882 - log message \"blah blah blah\"",
    "@timestamp" => 2019-04-29T18:47:46.679Z, 
           "msg" => ".9882 - log message \"blah blah blah\"",
       "logdate" => "19-04-29 12:47:46.679"
}

As you can see above, almost everything are correct on @timestamp
except the hours there
it suppose to be 12 instead of 18

in logentry timestamp is
"logdate" => "19-04-29 12:47:46.679"
but in @timestamp
"@timestamp" => 2019-04-29T18:47:46.679Z,

what am i doing wrong here,
I just need to have the timestamp from the log to be pushed out to ES....
Thanks a bunch in advance

2

elasticsearch stores timestamps as UTC. If you are in the US/Central timezone then it assumes your logs are too and adds 6 hours to get them to UTC. If your logs are already in UTC you can add a timezone option to the date filter to tell it that.

1 Like
3

@Badger

i added the timezone as below

input {
   stdin { }
}
filter {
    grok {
       match => { "message" => "%{NOTSPACE}%{TIMESTAMP_ISO8601:logdate}%{GREEDYDATA:msg}" }
    }
    date {
        timezone => "America/Edmonton"
        match => ["logdate", "YYYY-MM-dd HH:mm:ss.SSS", "ISO8601"]
    }
}
output {
    stdout { codec => rubydebug }
}

Still its not doing it right...

^X2019-04-29 12:47:46.679.9882 - log message "blah blah blah"
{
       "logdate" => "19-04-29 12:47:46.679",
          "host" => "thanu00-elk7-01",
       "message" => "^X2019-04-29 12:47:46.679.9882 - log message \"blah blah blah\"",
           "msg" => ".9882 - log message \"blah blah blah\"",
      "@version" => "1",
    "@timestamp" => 0019-04-29T20:21:38.679Z
}

I'm in Calgary/Edmonton timezone..

4

I was so confuse on date thing here is thread, and Badger help me understand it

https://discuss.elastic.co/t/date-to-cst/169615/3

5

Basically, eventho it shows the mytime +6 hours in my case, when it renders in Kibana or something it will show up in correct time ??
@elasticforme @Badger ^^^ ??

6

@Badger @elasticforme
Is this error familiar to you ?
image

7

Replace YYYY with YY

8

Yes, by default Kibana renders in the browser's timezone.

9

Please do not post pictures of text, just post the text. I am not familiar with that error.

1 Like
10

Thanks @Badger @elasticforme

got it resolved , This is all i needed....

    grok {
        match => {
            "message" =>
            "%{NOTSPACE}%{TIMESTAMP_ISO8601:logdate}%{NOTSPACE}%{GREEDYDATA:msg}"
            }
    }

    date {
        match => ["logdate", "yy-MM-dd HH:mm:ss.SSS", "ISO8601"]
    }

Thanks a lot for your prompt replies

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.