While overriding the @timestamp with log date , i am getting the time variation

Elastic Stack Logstash
1

Why i am getting the time variation while overriding the @timestamp with logdate

My input: INFO | jvm 1 | 2019/05/06 23:39:05.254 | FileName:xxxx_20190506 File Size:4096 File Collection time:2019-05-06 23:39:05.181
INFO | jvm 1 | 2019/05/06 23:39:05.254 | FileName:xxxx_1526726694_15_201904071415_201904071430.csv File Size:6897823 File Collection time:2019-05-06 23:39:05.183

my stdout:
filesize" => "13140176",
"message" => "INFO | jvm 1 | 2019/05/07 06:13:08.498 | FileName:xxxx_1526726660_15_201905061330_201905061345.csv File Size:13140176 File Collection time:2019-05-07 06:13:08.458",
"path" => "/usr/data/test.log",
"file_collection_time" => "2019-05-07 06:13:08.458",
"@timestamp" => 2019-05-07T10:13:08.498Z,
"host" => "localhost.localdomain",
"@version" => "1",
"logdate" => "2019/05/07 06:13:08.498",
"Filename" => "xxxxx_1526726660_15_201905061330_201905061345"

my config file:
input {
stdin{}
}

filter {

    if "pmexport" in [message] {

    grok {
            match => { "message" => "%{WORD:LogLevel}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE}%{SPACE}(?<logdate>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND})%{SPACE}%{NOTSPACE}%{SPACE}FileName:%{WORD:Filename}%{SPACE}File Size:%{INT:filesize}%{SPACE}File Collection time:%{TIMESTAMP_ISO8601:file_collection_time}"}

         }

    }

    else {

    grok {
             match => { "message" => "%{WORD}%{SPACE}%{NOTSPACE}%{SPACE}%{WORD}%{SPACE}%{NUMBER}%{SPACE}%{NOTSPACE}%{SPACE}(?<logdate>%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND})%{SPACE}%{NOTSPACE}%{SPACE}FileName:%{WORD:Filename}.%{WORD}%{SPACE}File Size:%{INT:filesize}%{SPACE}File Collection time:%{TIMESTAMP_ISO8601:file_collection_time}"}

        }

    }

    date {
    match => ["logdate","yyyy/MM/dd HH:mm:ss.SSS"]
    target=> "@timestamp"
    locale => "en"
    timezone => "America/New_York"
         }

        }

output {
stdout { codec => rubydebug }
}

2

logstash timestamps are in UTC. You have told the date filter that the log entries are in the timezone America/New_York, so it adds 4 hours to them (during DST) to get UTC.

3

so, now i have to replace the timezone as "UTC"

4

If i have single grok pattern and use timezone as "UTC" , then i am getting the correct value

5

Not sure what you mean by "the correct value". The entire elastic ecosystem assumes that timestamps are in UTC. So Kibana, for example, will translate those to the browser's timezone by default. If you are choosing to store timestamps in your local timezone you may find some things do not work as you expect.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.