GMT Timezone to CST in logstash

Elastic Stack Logstash
1

Hello,

Can anyone help me convert a GMT timezone to CST? I tried some options but haven't had accurate results so far.
I have a CSV file that contains fileds like below

"Record Type","Record Code","Broker Name","Broker UUID","EG Name","EG UUID","Message Flow Name","Message Flow UUID","Application Name","Application UUID","Library Name","Library UUID","Record Start Date","Record Start Time","Record GMT Start Timestamp","Record End Date","Record End Time","Record GMT End Timestamp","Total Elapsed Time","Average Elapsed Time","Maximum Elapsed Time","Minimum Elapsed Time","Total CPU Time","Average CPU Time","Maximum CPU Time","Minimum CPU Time","CPU Time Waiting for Input Messages","Elapsed Time Waiting for Input Messages","Total Number of Input Messages","Total Size of Input Messages","Average Size of Input Messages","Maximum Size of Input Messages","Minimum Size of Input Messages","Number of Threads in Pool","Time Maximum Number of Threads reached","Total Number of MQ Errors","Total Number of Messages with Errors","Total Number of Errors Processing Messages","Total Number of Time Outs Waiting for Replies to Aggregate Messages","Total Number of Commits","Total Number of Backouts","Accounting Origin"
"Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","00:52:31.599941","2017-02-16 06:52:31.5999","2017-02-16","01:46:17.773842","2017-02-16 07:46:17.7738","0","0","0","0","0","0","0","0","183935","3226073825","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous"
"Archive","Major Interval","DEV1","f328c29c-c695-11e5-addb-cc355a180000","PayoffQuote","9c276fa8-5201-0000-0080-941e766a88ad","com.payoffquote.PayOffQuote","273f59b2-5201-0000-0080-9c722b3eca55","","","","","2017-02-16","01:46:17.773949","2017-02-16 07:46:17.7739","2017-02-16","02:46:19.453657","2017-02-16 08:46:19.4536","0","0","0","0","0","0","0","0","193549","3601568195","0","0","0","0","0","1","0","0","0","0","0","0","0","Anonymous"

I am trying to get the Record GMT End Timestamp as my timestamp and convert it to CST.

add_field => {

"timestamp" => "%{Record GMT Start Timestamp}"}
remove_field => ["Record GMT Start Timestamp"]
}
date{
match => ["timestamp","yyyy-MM-dd HH:mm:ss,SSS"]
timezone => "America/Chicago"
remove_field => ["timestamp"]}
}

This is output i am getting off and i also get a

tags" => [
        [0] "_dateparsefailure"
    ]

What am i doing wrong? I can copy the whole config file if needed.

2

The timestamp field will contain "2017-02-16 06:52:31.5999" but your date pattern is "yyyy-MM-dd HH:mm:ss,SSS" so it clearly won't match.

Secondly, the date filter always converts to UTC. The timezone option sets the timezone of the input string, which in this case is GMT, so you should set timezone accordingly so it won't be assumed to be in local time.

(Side note: CST is ambiguous and could e.g. mean China Standard Time, Central Standard Time, or Cuba Standard Time.)

3

Yes. I realized the issue about the date pattern. I changed it to yyyy-MM-dd HH:mm:ss Z But i still get the same dateparsefailure error. Is there anyway i can simply remove the millisecs or microsecs?
Re: the timezone, i am trying to set Central Standard Time. How would i set the timezone as you suggested so it won't assume to be in local time?

4

Yes. I realized the issue about the date pattern. I changed it to yyyy-MM-dd HH:mm:ss Z But i still get the same dateparsefailure error.

But that pattern also doesn't match "2017-02-16 06:52:31.5999". I'd try yyyy-MM-dd HH:mm:ss,SSSS.

Is there anyway i can simply remove the millisecs or microsecs?

Sure, you can use the mutate filter's gsub option.

Re: the timezone, i am trying to set Central Standard Time. How would i set the timezone as you suggested so it won't assume to be in local time?

I think a list of available timezone name are available via a link in the docs. I suspect that timezone => "Etc/UTC" might work for you.

5

Thanks a lot for the responses Magnus. But i tried Etc/UTC and i get a revised timezone but it is off by 12 hours. So when i should get 08:46 I get 20:46. I tried other timezones listed, America/Chicago, Etc/GMT+6 etc. But none seem to give the exact timezone.

6

This works as expected for me:

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  date {
    match => ["message", "yyyy-MM-dd HH:mm:ss.SSSS"]
    timezone => "Etc/UTC"
  }
}
$ echo '2017-02-16 06:52:31.5999' | /opt/logstash/bin/logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "2017-02-16 06:52:31.5999",
      "@version" => "1",
    "@timestamp" => "2017-02-16T06:52:31.599Z",
          "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
7

But that's the thing, I am trying to change the time in your echo to Central Standard Time. Since my input for timestamp will always be in GMT. so, if the echo is 06:52:31.5999 supposed it is GMT, my output @timestamp should show "@timestamp" => "2017-02-16T00:52:31.599Z",

8

As I said, the date filter always converts to UTC. Hence, you can't configure it to output UTC-5.

9

Got it. Is there anyway i can achieve that? Take my GMT time from CSV and convert it to CST timestamp?

10

Actually, i worked around this. instead of taking the GMT column from the CSV, i am taking Record End Date and Record End Time which are in CST, combining them and then getting the timestamp. It's giving me the desired output. And this should work well enough. Once again, thanks a ton for the support Magnus.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.