Syslog input to logstash time zone - drifts by 2h. everything set to UTC

Elastic Stack Elasticsearch
1

Debug stdout

logstash    | {
logstash    |        "message" => "notice syslog-ng[2219]: Syslog connection established; fd='67', server='AF_INET(10.6.10.93:5140)', local='AF_INET(0.0.0.0:0)'\n",
logstash    |          "event" => {
logstash    |         "original" => "<45>Jul 10 17:01:40 bigip-f501.core.company.net notice syslog-ng[2219]: Syslog connection established; fd='67', server='AF_INET(10.6.10.93:5140)', local='AF_INET(0.0.0.0:0)'\n"
logstash    |     },
logstash    |       "@version" => "1",
logstash    |     "log_source" => "bigip_syslog",
logstash    |     "@timestamp" => 2024-07-10T15:01:40.000Z,
logstash    |           "host" => {
logstash    |         "hostname" => "bigip-f501.core.company.net",
logstash    |               "ip" => "10.5.10.5"
logstash    |     },
logstash    |            "log" => {
logstash    |         "syslog" => {
logstash    |             "facility" => {
logstash    |                 "code" => 5,
logstash    |                 "name" => "syslogd"
logstash    |             },
logstash    |             "priority" => 45,
logstash    |             "severity" => {
logstash    |                 "code" => 5,
logstash    |                 "name" => "Notice"
logstash    |             }
logstash    |         }
logstash    |     },
logstash    |        "service" => {
logstash    |         "type" => "system"
logstash    |     }
logstash    | }

Current date on my devices are also in UTC (currently ~5pm)
I am running logstash via docker-compose.
Docker and local host has set time to UTC.

sliddjur@logstash:/opt/logstash$ docker exec -it logstash date
Wed 10 Jul 2024 05:02:28 PM UTC
sliddjur@logstash:/opt/logstash$ docker exec -it logstash env | grep TZ
TZ=UTC
sliddjur@logstash:/opt/logstash$ date
Wed Jul 10 17:04:18 UTC 2024

My bigip f5 device is sending syslog. The f5 is configured with time zone UTC. As you can see, the original log message does not include timezone, so logstash should assume UTC?:
logstash | "original" => "<45>Jul 10 17:01:40 bigip-f501.core.company.net notice syslog-ng[2219]: Syslog connection established; fd='67', server='AF_INET(10.6.10.93:5140)',

However, the debug shows that the @timestamp field is set to 2024-07-10T15:01:40.000Z - so somewhere it has drifted two hours.

I was changing from Europe time zone on logstash, both on logstash server and docker environment. I have restarted container since then, but I have no clue why logstash still converts logs two hours back.

The elastic cluster is configured with Europe time zone locally. Does logstash know the time of the elastic cluster and changes the @timestamp field before it shows stdout debug? Seems unlikely though, as I am getting the same result without elasticsearch output, and only debug output.

input {
  syslog {
    port => 5140
    add_field => { "log_source" => "bigip_syslog" }
  }
}

output {
  if [log_source] == "bigip_syslog" {
    stdout {
      codec => rubydebug
    }
  }
}
2

Hi,

try adding timezone

input {
  syslog {
    port => 5140
    add_field => { "log_source" => "bigip_syslog" }
    timezone => "Europe/Stockholm" # adjust this to your actual timezone
  }
}

Regards

3

How were you changing this?

Can you share your entire pipeline, not just this snippet? Do you have any date filter later ?

If your source was already in UTC and you didn't had any timezone in the syslog input, I would not expect Logstash to offset your date.

5

How were you changing this?

This is a Ubuntu system, so I was specifically changing with sudo timedatectl set-timezone UTC.
My docker-compose with environment variable TZ=UTC. Confirmed above in first post with date output.

If your source was already in UTC and you didn't had any timezone in the syslog input, I would not expect Logstash to offset your date.

Yes I agree, that is why I am confused aswell. I am not using date filter.

I am specifically looking at bigip_syslog that inputs on port 5140. Here is my full pipeline.

input {
  syslog {
    port => 5140
    add_field => { "log_source" => "bigip_syslog" }
  }
  udp {
    port => 5141
    codec => json
    add_field => { "log_source" => "bigip_accesslog" }
  }
}


filter {
  if [log_source] == "bigip_accesslog" {
    geoip {
      source => "[source][ip]"
      database => "/usr/share/logstash/geolite2/GeoLite2-City.mmdb"
    }
    mutate {
      copy => { "[source][ip]" => "[source][domain]" }
    }
    dns {
      reverse => ["[source][domain]"]
      action => "replace"
      nameserver => {
        address => [ "10.6.10.10" ]
      }
      failed_cache_ttl => 300
      hit_cache_ttl => 300
      max_retries => 1
      timeout => 0.5
    }
  }
}


output {
  if [log_source] == "bigip_syslog" {
    stdout {
      codec => rubydebug
    }
    elasticsearch {
      hosts => ["company-elastic01:9200", "company-elastic02:9200", "company-elastic03:9200", "company-elastic04:9200"]
      index => "network-syslog-%{+YYYY-MM-dd}"
    }
  }
  else if [log_source] == "bigip_accesslog" {
    elasticsearch {
      hosts => ["company-elastic01:9200", "company-elastic02:9200", "company-elastic03:9200", "company-elastic04:9200"]
      index => "network-f5accesslog-%{+YYYY-MM-dd}"
      ecs_compatibility => "v8"
    }
    #stdout {
    #  codec => rubydebug
    #}
  }
}
6

This is the only config file that you are running or may other configs exist?

I see nothing in it that could cause this shift, maybe it is a bug and opening an issue would be needed.

Have you tried to for the timezone in the syslog input with timezone => "UTC".