Timestamp incorrect, even in regards to UTC

Elastic Stack Elasticsearch
1

Hi, I am using the elk stack in regards to netflow records (using the netflow logstash codec). I have noticed a strange discrepancy regarding the timestamp. I'm aware elasticsearch converts time to UTC, but my issue seems to go beyond this. For example, this just happened:

UTC time is 6:15 pm, and my local time is 11:15 am (pacific time)..
-My logstash output that's being dumped to the terminal has a timestamp of 11:07am
-The timestamp that is available in elasticsearch/kibana discover tab is 4:07am

So not only is it not aligned with UTC hours-wise, but it is also off by 8 minutes. If anyone has any ideas on how to fix this, I really appreciate it.

2

The Pacific time zone (typically -8) is currently in Daylight Saving Time, meaning it is currently -7. This seems to explain the 11AM logstash stdout vs 4AM Kibana browser display. Logstash is displaying the time in UTC on stdout, and Kibana is receiving the UTC time and applying your local system offset (-7).

On an incoming timestamp to the logstash date filter, if the timezone is not detected - it will apply the local offset (-7). https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-timezone. This would explain 6 PM - 7 hours = 11 AM.

As far as the 8 minute difference, this is most likely time not being synchronized properly on one of the devices in question.

3

Perhaps the confusion on my part, but logstash is not displaying the time in UTC on stdout. UTC would be 6:15pm in this case. It displays 11:07am, aka more closely aligned with my local Pacific Time. Furthermore, why would Kibana offset it further (subtract another 7 hours) from UTC? If the name of the game is to align it with UTC it should be adding 7 hours to make it 6pm, correct?