Wrong timestamp, when trying to convert local time to UTC

[asmerb (Asmer Bracho)]

Hi everyone,

I am using LogStash and ElasticSearch 6.3.1

Currently having an issue where my @timestamp is being shifted by two hours.

I am picking up records in logstash that comes from filebeat, the timestamp here is being recorded in local time (Europe/Vienna which is UTC +2).

I understand logstash records time in UTC and if I am not misunderstanding for the reading and research, this is where the 2 hours get shifted. So I am trying to tell logstash that the time is in local time Vienna (UTC +2) so logstash can store the time correctly.

I am ussing the folllowing configuration and still getting the 2 shifted hours, Hoping someone can give a hand to understand what the problem can be.

Using a grok filter to match the message as follow:

grok {
match => { "message" =>
"%{INT:timestamp},%{DATA:transactionId},%{WORD:eventType},%{INT:subscriberId},%{DATA:action},%{DATA:accountId},%{DATA:accType},%{DATA:socName},%{DATA:subsoc},%{DATA:socList},%{GREEDYDATA:subTariff},%{GREEDYDATA:socDate},%{DATA:errorCode},%{GREEDYDATA:errorMessage},%{GREEDYDATA:status},%{DATA:subscriberRole},%{DATA:accTariff},%{DATA:cycleStartDay},%{DATA:accEmail},%{DATA:SOCDealerCode}"
}

Then a ruby filter that takes the timestamp an create a firl CREATED_ON_DATE that contains the timestamp as UNIX

 ruby {
 init => "require 'date'"
 code => "event.set('CREATED_ON_DATE',DateTime.parse(event.get('timestamp')).to_time.to_i)"
}

and finally a date filter to match CREATE_ON_DATE to @timestamp:

date {
match => ["CREATED_ON_DATE", "UNIX"]
timezone => "Europe/Vienna"
target => "@timestamp"
}

See the diferent on the timestamp (+2 hours)

Thanks in advance for the support

[Jenni]

I don't think that your timezone setting will do anything if your date is already a unix timestamp and not a formatted string anyway. The problem is that your Ruby code already interprets your date as UTC (as you can see if you convert the integer value that it has created). Is there a reason for that intermediate step? Couldn't you parse it with a date filter with yyyyMMddHHmmss (and with the timezone setting because then this should take effect)?

[asmerb (Asmer Bracho)]

Hi Jenni, Thanks for your quick response, I see the problem now, and indeed you are 100% correct, I have done what you suggested and it is working fine, so I will go with that aproach , thanks again for your support.

 date {
                     match => ["timestamp", "yyyyMMddHHmmss"]
                     timezone => "Europe/Vienna"
                     target => "@timestamp"
                 }