I have a strange case where a Docker container (system time runs in timezone UTC) sends HAProxy logs to a remote syslog server and from there into logstash.
On the remote syslog server the log arrives:
Aug 25 11:38:00 192.168.40.15 haproxy[23540]: 10.10.10.11:52596 [25/Aug/2017:11:38:00.017] 80 80_enrico_test_/a1b8fdd53241fe9032c1b73ce55292db0e7fe082 0/0/1/2/3 200 389 - - ---- 11/11/0/0/0 0/0 "GET /_health HTTP/1.1"
The syslog server is running in UTC+2 (therefore 2h ahead).
Now when logstash (on the syslog server) gets this message it (probably) thinks:
"Hey I'm running 2 hours ahead but I want to store times in UTC. To do that I have to take the timestamp and remove 2 hours".
Which is OK for all logs arriving in UTC+2 but not the ones which arrive in UTC on the syslog server. Because when logstash handles this log entry coming from the Docker container, it results in a time difference of 4 hours:
Aug 25 13:38:00 inf-elk01-t logstash[8657]: {
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "severity" => "info",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "@timestamp" => 2017-08-25T09:38:00.000Z,
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "@version" => "1",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "programname" => "haproxy",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "procid" => "23540",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "host" => "0:0:0:0:0:0:0:1",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "sysloghost" => "192.168.40.15",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "message" => " 10.10.10.11:52596 [25/Aug/2017:11:38:00.017] 80 80_enrico_test_/a1b8fdd53241fe9032c1b73ce55292db0e7fe082 0/0/1/2/3 200 389 - - ---- 11/11/0/0/0 0/0 \"GET /_health HTTP/1.1\"",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "type" => "rsyslog",
Aug 25 13:38:00 inf-elk01-t logstash[8657]: "facility" => "local0"
Aug 25 13:38:00 inf-elk01-t logstash[8657]: }
Take a look at @timestamp and you know what I mean.
Any idea how this can be handled properly?
I already looked at other discussions here (e.g. Dealing with multiple timezones) but without having found a solution.
Update August 28th: I was able to solve this by using the "date" function within the filter for HAProxy. See https://www.claudiokuenzler.com/blog/730/handling-different-timezones-haproxy-logs-elk-logstash-stack for details.