Doubled time zone in Logs

[fetch (Ivan)]

Hello,

ELK versions: 6.8.0

I've just enabled "Logs" feature in Kibana and came across issue when timezone value is doubled.

I'm using filebeat to send Elasticsearch's own logs. filebeat's config:

- module: elasticsearch
  # Server log
  server:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  gc:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

  audit:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

  slowlog:
    enabled: true
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  deprecation:
    enabled: false
    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    #var.paths:

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    #var.convert_timezone: false

When I check events in Kibana it says the following:

beat.timezone +03:00
event.created 2019-05-30T06:23:34.308Z
@timestamp 2019-05-30T09:23:32.467Z

However, on the Logs tab the time displayed is +3 hours from the value in @timestamp field - 2019-05-30 12:23:32.467

On the Management -> Advanced tab time zone is set to "Browser". I've tried setting it to UTC, but it had no effect.

Any ideas how to fix it?

[Bargs (Matt Bargar)]

@timestamp is in UTC, so it is expected that Kibana will convert this into your local timezone in the UI. (It's unusual that switching to UTC had no effect. I just tested this and it worked. Did you reload the page after changing the setting?)

However, I suspect the issue is in the timezone conversion happening in Beats/ES, not in Kibana. What is the actual timestamp listed in the Elasticsearch log file? I am guessing it does not match the value of @timestamp that you included above. The reason I say this is that the @timestamp you provided occurs after event.created. event.created represents the time that Filebeat read the log, whereas @timestamp is the time from the log itself, so event.created should always be after @timestamp.

[fetch (Ivan)]

Hi Bargs,

Thanks for the reply. I've collected some more info to compare values.

The message from ES's log file stored on the host:

[2019-05-31T16:36:10,164][INFO ][o.e.c.m.MetaDataMappingService] [elk2] [files/0gkSsbzyTAmeb2-bleCkPQ] update_mapping [doc]

In Kibana->Logs it's listed as:

2019-05-31 19:36:10.164 [elasticsearch.server][INFO] update_mapping [doc]

When I click details for the event in kibana it has the following values:

@timestamp 2019-05-31T16:36:10.164Z
beat.timezone +03:00
event.created 2019-05-31T13:36:18.365Z

"Timezone for date formatting" in Kibana->Advanced settings is set to "UTC" (I've opened the Logs tab in incognito mode after altering this setting).

So the @timestamp listed in event details is correct (GMT+3) and matches timestamp listed in ES's log file (ES host has timezone GMT+3). event.created timestamp looks good too since filebeat have "convert to utc" enabled. However, Kibana still shows timestamp is +3 hours from it's real value.

Maybe the change in "Timezone for date formatting" applies only after kibana's restart?

[Bargs (Matt Bargar)]

@timestamp 2019-05-31T16:36:10.164Z

This tells me that @timestamp in the Elasticsearch document is set to 16:36:10 UTC. If I'm understanding you correctly, this time is incorrect. It should be 13:36:10 UTC. So the time stored in the Elasticsearch document is already incorrect, before it ever gets to Kibana. I assume this is because the timestamp in the Elasticsearch log does not have timezone information, so when it gets indexed ES assumes the time is in UTC.

If you agree with all that, I can move this thread over to the Beats forum where they can probably show you the correct way to fix this.

[fetch (Ivan)]

Okey, if it's the filebeat's issue, please move it to Beats forum.

Thanks.