Logstash/kibana timezone issue

[edvrfn (edv)]

Below is the sample log message

<12>May 5 00:10:28 UDM,50c9676c6d24,udm-1.9.3.3438 kernel: [522844.682013] IN=br30 OUT=eth4 MAC=76:ac:b9:1e:f2:fc:dc:a6:32:eb:22:ac:08:00 SRC=192.168.3.53 DST=132.145.110.200 LEN=61 TOS=0x00 PREC=0x00 TTL=63 ID=61244 DF PROTO=UDP SPT=45838 DPT=53 LEN=41

But Kibana displays it as below with a 2 hour time difference after changing the timezone to America/Vancouver in Index management > advanced settings. Changing it to any other timezone just shifts the time but histogram remains 2 hours behind.

timezone of system

[root@sof-elk filebeat]# timedatectl
      Local time: Tue 2021-05-04 22:38:00 PDT
  Universal time: Wed 2021-05-05 05:38:00 UTC
        RTC time: Wed 2021-05-05 07:26:22
       Time zone: America/Vancouver (PDT, -0700)

how can I fix this. Also logstash seems to make new index file in UTC

[matw (Matthias Wilhelm)]

Hi

The time in the index should be UTC, but in your case, maybe the value is wrong. There is a delta of 2 hours Universal time vs RTC time in your system, so I wonder If RTC time is written into the index, assuming it is UTC time. That would explain the 2 hours delta. If the UTC time written is correct, also Kibana would display it correctly.

Best,
Matthias

[edvrfn (edv)]

@matw If I change the timezone in kabana to UTC, the time is shown in UTC but now the delta increases to 4. Any clue what/where should I be looking for.

[matw (Matthias Wilhelm)]

yes, changing timezone in Kibana won't help, you should take care that the time that is set in logstash and sent to ES is UTC. And not just the format of UTC, also the value has to be correct. If you index a value that's correct for another timezone as UTC time, you will get a gap. And I'm sure this is the case in your scenario. Have a look at the raw values of you ES data.

Best,
Matthias