Add_locale is inserting wrong timezone

[John_Lofgren]

I am using filebeat -> logstash -> elasticsearch.

I am trying to harvest a log file that uses timestamps in the machine's local timezone, but without any explicit indication of what the timezone is. Example:

[2018-01-26 13:31:21.331]...

My servers are in many different timezones, so I need to normalize these times. I think the new add_locale processor is supposed to help with this by adding the machine's timezone into the output. But, the timezone is always reported as "+00:00" in the output.

My filebeat config includes:

processors:
- add_locale: ~

Here is an example rubydebug output from logstash:

"@timestamp" => 2018-01-26T19:33:30.733Z,
    "offset" => 9195,
  "@version" => "1",
    "format" => "...",
      "beat" => {
        "name" => "...",
    "hostname" => "...",
     "version" => "6.1.2",
    "timezone" => "+00:00"
},
      "host" => "...",
"prospector" => {
    "type" => "log"
},
    "source" => "...",
   "message" => "[2018-01-26 13:33:22.064]...",
      "tags" => [
    [0] "beats_input_codec_plain_applied",
    [1] "_grokparsefailure"
]

The system reports the correct timezone from the date command:

$ date +'%Z %z'
CST -0600

Am I doing this right? By what means does add_locale determine the timezone of the machine?

[andrewkroh]

What OS? How did you start the Beat? Was the Beat executed in the same environment that you ran date from?

On Unix the timezone is determined with:

	// consult $TZ to find the time zone to use.
	// no $TZ means use the system default /etc/localtime.
	// $TZ="" means use UTC.
	// $TZ="foo" means use /usr/share/zoneinfo/foo.

[John_Lofgren]

This is my development environment where this is occurring: CentOS 7 running in Docker on a CentOS 7 VM, running in VMware on a Windows 10 host. So - CentOS 7.

filebeat is just started from the command line:

/path/to/filebeat/bin/filebeat -e -c /path/to/filebeat/filebeat.yml >> /path/to/filebeat/log/filebeat.log 2>&1

Yes, I ran the date command in the same docker container.

$TZ seems to work correctly also:

$ echo $TZ
/usr/share/zoneinfo/America/Chicago

[andrewkroh]

So this filebeat instance is running inside of a Docker container? What image are you running?

[John_Lofgren]

It is built from centos latest: https://hub.docker.com/_/centos/

To that we add the following packages along with some other minor tweaks, none of which I think should have any bearing on the issue:

bind-utils dnsmasq emacs-nox git htop \
net-tools openssh-server openssl \
psmisc python2-pip screen snappy snappy-devel \
sudo the_silver_searcher tree which wget

[John_Lofgren]

I think we've figured out the cause. In the docker image we work with, the TZ variable was set in the .bashrc as:

export TZ="/usr/share/zoneinfo/America/Chicago"

So, apparently add_locale was looking for a timezone spec at /usr/share/zoneinfo/usr/share/zoneinfo/America/Chicago? And when it didn't find that, I assume it reverted to the system default or to UTC.

When I changed to:

TZ="America/Chicago"

filebeat worked correctly! (And the date command still does, too.)

This discussion got me to read 'man timezone' on CentOS 7. There are a whole lot of other valid formats of TZ that appear to be supported by the OS, so something like this is likely to come up again. Curiously - man timezone indicates that the filespec format is supposed to start with a colon ':' followed by either a relative path or absolute path to the timezone spec file. But, apparently the colon is optional for some tools!

I suggest at least documenting on the add_locale page the TZ algorithm you described above. And you may consider adding support for more formats.

Thanks!

[andrewkroh]

I'm really glad you figured out the root cause. BTW the TZ algorithm is entirely implemented as part of the Go stdlib, and I just copied their pseudo code comments from the source.