@timestamp converts timezones when it shouldn't - Weird issue

Elastic Stack Logstash
1

Hello,

I'm new to Elastic Search Logstash and Kibana and would appreciate some help with an issue I've been unable to resolve.

I'm forwarding logs from various labs in Canada (EDT), Turkey (EEST), and Vietnam (IST) using filebeat and am having a strange conversion happen to @timestamp when i try and view it in Kibana. I have logstash installed on servers local to me in Canada.

I need a way to be able to visualize what labs are receiving logs in real-time, i.e a parameter which captures the time of filtering.

It is my understanding that @timestamp is supposed to save the time that logstash filters the incoming logs. However depending on which time zone the log is coming from it seems to shift @timestamp as if trying to convert from the source location to Ottawa's timezone even though i beleive it is already in EDT.

I've defined a parameter called source-timestamp which is the time the log-line was written (taken from timestamp in log). This is also required for analysis.

Here is a screenshot of some logs that have been sent from Turkey (7 hours ahead of my local time)

Pasted image1410×400 19.5 KB

as you can see Kibana has converted the source-timestamp (from the log lines) into my time for me but @timestamp has been shifted as well even though it shouldn't be.

When a log is from Vietnam it has the same behaviour except it's trying to shift @timestamp from IST to EDT.

Here is the date filter i use to set "source-timestamp"

date {
match => [ "Date", "MMM dd HH:mm:ss", "MMM dd HH:mm:ss", "MMM d HH:mm:ss",
"ISO8601", "YYYY-MM-dd HH:mm:ss", "YYYY.MM.dd HH:mm:ss", "YYYY-MM-dd HH:mm:ss,SSS",
"YYYY/MM/dd HH:mm:ss.SSS", "EEE MMM dd HH:mm:ss z yyyy", "MM/dd/YY-HH:mm:ss",
"MMM dd, YYYY KK:mm:ss aa", "dd/MMM/YYYY:HH:mm:ss Z", "YYYY-MM-dd HH:mm:ss:SSSS",
"YY.MM.dd HH:mm:ss.SSS", "EEE MMM dd HH:mm:ss YYYY", "YYYY-MM-dd HH:mm:ss.SSSSSS",
"HH:mm:ss MM/dd/YYYY", "EEE MMM d HH:mm:ss z yyyy", "EEE MMM dd HH:mm:ss ZZZ YYYY",
"dd-MMM-YYYY HH:mm:ss.SSS", "dd MMM YYYY HH:mm:ss:SSS", "dd MMM YYYY HH:mm:ss,SSS",
"EEE MMM dd HH:mm:ss ZZ yyyy", "dd/MMM/YYYY HH:mm:ss Z" ]
target => "source-timestamp"
}

Date is the time-stamp captured from the log-lines.

Thank you in advance for any guidance you could give me.

Thomas

2

The raw @timestamp value is supposed to be UTC. By default Kibana adjusts that field for the timezone of your browser. Doesn't that explain what you're seeing?

It is my understanding that @timestamp is supposed to save the time that logstash filters the incoming logs.

@timestamp should be the time when the event occurred. When Logstash processes an event is usually not important.

3

I should be seeing that source-timestamp and @timestamp are in the same hour..... It looks like @timestamp has been shifted 7 hours (EEST to EDT) when it shouldnt have..... The same happens with an 11 hour difference when the source lab is in Vietnam (ICT to EDT). This makes it so when i try to visualize labs from multiple time zones on Kibana they do not align on the current time (they are offset by however many hours).....

4

In your source logs, what is the timezone used to log date ? local timezone ? or GMT timezone ?

I invite you to consider "timezone" option (in date filter) to specify which is the default timezone when date is parsed.

5

The source timestamp are always written in their local time however for some log files the timestamp does not specify the timezone that it is in (frustrating but i cannot change this). I understand when there is no timezone in the timestamp that it should be specified in the date filter using the "timezone" option.

But the value I'll need to specify for "timezone" option will change depending on which lab it's coming from (Turkey / Vietnam / Ottawa).

How can I make the filter handle this type of situation?

Thanks.

6

I suppose you have (at least) a logstash instance at each local place, so I advice you to :

  • explicitly set export TZ=TheLocalTimeZone just before running logstash
  • and set "date" filter on local logstash instance (if you use a centralized logstash instance, do not parse date on central instance, if possible)
7

We actually use filebeat to forward logs to a centralized logstash instance, there are no such local instances of logstash for each lab.

Would it be possible to use the date filter and assign a timezone based on the IP address? or by Lab name?

ex:
filter{
if "vietnam location defining attribute" in [source-ip] / [lab-name] {
date {
match => ......
target => "source-timestamp"
timezone => "IST"
}
}
else if "Turkey location defining attribute" in [source-ip] / [lab-name]{
.....
.....
timezone >= "EEST"
}
....
}

8

With recent versions of the date filter it's possible to reference other fields in the timezone setting, so you can set a field like [@metadata][timezone] based on the IP address or whatever and then do this:

date {
  match => ...
  timezone => "%{[@metadata][timezone]}"
}
9

Great that looks like it'll solve my problem, thanks!

One last question:
Is it a problem if a timestamp already contains the timezone and timezone is being assigned in this way in the date filter? I only ask because some of the timestamp formats for logs coming in already contain a timezone.

Thanks!

10

I'm pretty sure the timezone in the text being parsed will be honored if your pattern contains a %z token.

11

Concerning getting timezone from source machine, you can add a "timezone" field in beats configuration, using this :
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#configuration-fields

The timezone field could contain directly the value (ex: Europe/Paris), or a reference to an environment variable : ${TZ}

Then in logstash, you use it in date filter :
date {
timezone => "%{timezone}"
}

12

Thank you for your help! Assigning a timezone field fixed the issue!