Daylight saving time impacting only part of the environment

Elastic Stack Logstash
1

Hello everyone, we are facing a strange behavior in ELK environment related to daylight saving time; the details are below:

Platform:

  • kibana 6.7.2
  • Logstash 5.3.0
  • Elasticsearch 6.7.2

Scenario:

  • We have Weblogic domain groups that have their logs available graphically in Kibana - Groups A, B, C and D;
  • In Brazil there is no more daylight saving time since 2019.

Problem:

  • Always occurs in the period that would be the first day of daylight saving time and ends after the finalization period;
  • Group D shows the graph delayed by 1h;
  • Groups A, B and C show graphics normally with normal time.

Actions already taken, but unsuccessful:

  • Checked timeZone in Logstash "apache-reader.conf" file and all show tmeZone as "America/Sao_Paulo" / "GMT -3"
  • Checked in Kibana, "dateformat:tz" is like "browser" ; when it changes to "America/Sao_Paulo", it corrects it to normal time (group D) and advances the others by 1h (Groups A, B and C).

Regards,
Paulo Santos

2

What do you mean by group A, B, C and D? These are different servers? What are their timezones?

What is the time in the log file you are parsing? Share an example.

What is the configuration of your date filter in your Logstash configuration?

What is the timezone configuration of your server, are they in UTC or in the local timezone? Also, are they Linux or Windows?

Are the servers updated?

3

Hello Leandro,

Follow the answers, let me if you need more information.

++++++++++++++++++++++++++++++++

  • What do you mean by group A, B, C and D?
    I only named by groups for a better understanding, but we can figurate them like this:
    Group A - JBoss app for specifi funcion - (oK in Kibana)
    Group B - Weblogic specific domains group app for specifi funcion - (oK in Kibana)
    Group C - Weblogic specific domains group app for specifi funcion - (oK in Kibana)
    Group D - Weblogic specific domains group app for specifi funcion <-- this the environment that has the issue (1h delayed) to show graphics in Kibana

  • These are different servers?
    Yes

  • What are their timezones?
    App servers per group:
    The timeZones in the these App servers(A, B, C and D) is "/usr/share/zoneinfo/America/Sao_Paulo"
    Group A - /usr/share/zoneinfo/America/Sao_Paulo
    date "+%z%Z"
    -0300-03

Group B - /usr/share/zoneinfo/America/Sao_Paulo
Local time: Mon 2021-11-08 13:33:51 -03
Universal time: Mon 2021-11-08 16:33:51 UTC
RTC time: Mon 2021-11-08 16:33:51
Time zone: America/Sao_Paulo (-03, -0300)
NTP enabled: no
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a

Group C - /usr/share/zoneinfo/America/Sao_Paulo
date "+%z%Z"
-0300-03

Group D - /usr/share/zoneinfo/America/Sao_Paulo
date "+%z%Z"
-0300-03

ELK logstash servers --> it is in UTC, as it shown below:
Local time: Mon 2021-11-08 10:37:56 -03
Universal time: Mon 2021-11-08 13:37:56 UTC
RTC time: Mon 2021-11-08 13:37:56
Time zone: America/Sao_Paulo (-03, -0300)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a

  • What is the time in the log file you are parsing? Share an example.
    Folow the examples:

Group A -
JVM_accesslog --> 08/Nov/2021:00:00:01 -0300
server.log --> 2021-11-08 12:00:06,200

Group B -
JVM_acess_log --> 08/Nov/2021:12:11:29 -0300
JVM_log --> <07/11/2021 10h53min47s GMT-03:00> and <07/11/2021 10h51min32s BRT>

Group C -
JVM_acess_log --> 2021-11-07 11:00:14
JVM_log --> <Nov 7, 2021 10:57:29 AM BRT>

Group D -
JVM_access_log --> 2021-11-08 11:04:36
JVM_log --> <08/11/2021 10h06min34s BRT>

What is the configuration of your date filter in your Logstash configuration?
ELK logstash filter:
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
timezone => "America/Sao_Paulo"
}

  • What is the timezone configuration of your server, are they in UTC or in the local timezone?
    Group A - /usr/share/zoneinfo/America/Sao_Paulo
    date "+%z%Z"
    -0300-03

Group B - /usr/share/zoneinfo/America/Sao_Paulo
Local time: Mon 2021-11-08 13:33:51 -03
Universal time: Mon 2021-11-08 16:33:51 UTC
RTC time: Mon 2021-11-08 16:33:51
Time zone: America/Sao_Paulo (-03, -0300)
NTP enabled: no
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a

Group C - /usr/share/zoneinfo/America/Sao_Paulo
date "+%z%Z"
-0300-03

Group D - /usr/share/zoneinfo/America/Sao_Paulo
date "+%z%Z"
-0300-03

ELK logstash servers --> it is in UTC, as it shown below:
Local time: Mon 2021-11-08 10:37:56 -03
Universal time: Mon 2021-11-08 13:37:56 UTC
RTC time: Mon 2021-11-08 13:37:56
Time zone: America/Sao_Paulo (-03, -0300)
NTP enabled: yes
NTP synchronized: yes
RTC in local TZ: no
DST active: n/a

  • Also, are they Linux or Windows?
    Group A - Linux RHEL AS 4 / Linux RHEL 5.11
    Group B - Linux RHEL 7.4 / Linux RHEL 5.11
    Group C - Linux RHEL 5.11
    Group D - Linux RHEL 5.11
    ELK logstash serves - Linux CentOS 7.7.1908

  • Are the servers updated?
    If you mean about the last O.S. and/or ELK version, they are not.

4

You might want to verify that the Olson/zoneinfo/tz database for America/Sao_Paulo actually reflects that on each group.

5

Can you make a little test, change timezone to "-0300" instead of America/Sao_Paulo?

I had a similar problem when the Daylight saving time was abolished in Brazil in 2019, while the server was at the correct time and the tzdata packages were updated, this makes no difference as the internal library used by logstash to parse the data was not updated.

You also could try America/Araguaina instead of America/Sao_Paulo, it is a location that never had Daylight saving time and was always in UTC-0300.

1 Like
6

Hello Leandro,

It´s solved after your guide in changing the timezone in logstash configuration !

Now the "Group D" has the same date/time that Groups A,B and C and they appear normally in Kibana.

Thank you !

Paulo Santos

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.