Filebeat and Timezones

Elastic Stack Beats
1

Full disclosure, just getting started with my first ELK stack setup...

When I use Filebeat to forward syslogs I have some hosts in UTC and some in EST. Once they go into Elastic Search it assumes everything is in UTC which puts timestamp searching out by a few hours. (viewed through Kibana)

Is there a way to configure the Filebeat agent so that it will convert the dates into UTC before submitting to LogStash/Elastic Search? Or is there a preferred way to address this?

2

do you index right into elasticsearch or via logstash?

filebeat just collects log lines and reports times a line has been read in UTC already. But content of line is not adjusted. If you're using any kind of filter 'grok' in logstash to parse line and timestamp, the timestamp adjustment must be made in logstash.

Maybe you can use beat.hostname, beat.name, type or any other information to filter the source.

3

Hi Steffens

I am loading the data via logstash. I'm using the docker appliance sebp/elk. Below is a copy of the syslog.conf on the logstash server.

So I'm gathering that I need to define a new filter type on the logstash server i.e. "syslog_est" and then set the timestamp as EST in that filter? If so, what is the proper syntax of defining the timezone?

##############################
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

4

For anyone who should come across this, here's how I solved it:

  1. I created a copy of /etc/logstash/conf.d/10-syslog.conf as it /etc/logstash/conf.d/11-syslog.conf on the logstash server.
  2. Edit the new conf file and change if [type] == "syslog" to if [type] == "syslog_est"
  3. Again edit the new conf file and edit the date directive to include the timezone:

date {
timezone => "America/New_York"
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}

Edit the /etc/filebeat/filebeat.yml on the filebeat agent to:

filebeat:
prospectors:
-
paths:
- /var/log/secure
- /var/log/messages
document_type: syslog_est
-

Restart the filebeat service and the log timestamps in elastic search are now correct.

Alternatively, if you can configure your ssh service which writes to /var/log/secure to use UTC, this might be the better way to go.

5

Cool. If possible you should try to normalize time zones at source or include timezone info in your original logs.