Time incorrect from system module (syslog)

Elastic Stack Beats
1

Beats and ES 6.2.2

When we enable the System module the time of syslog entries in Kibana shows as UTC appearing 5 hours old. The syslog entries themselves have the correct EST time.

If we enable log watching in the filebeat.yml for *.log in var/log/ the times of say UFW and Auth show correctly in Kibana.

We are not doing any time manipulation with the filebeat.yml or the system.yml file.

2

Elasticsearch stores all "time" fields in UTC/Epoch

a couple things could be going on.

The simplest of them. Kibana uses the Timezone of your browser to display the timezone.

Are you talking about the _timestamp field or your own field?

Is that field of type string or time?

If that does help to answer your problem please provide some of the json data from Elastic so we can see what your talking about

3

Kibana is using browser timezone and logs like ufw and auth to show correctly in Kibana.

When viewing the timeline in Kibana the current syslog logs (system module) being displayed are 5 hours behind as opposed to the auth and ufw logs (configured in filebeat.yml) which are being written at the current time.

We're not using custom fields. Everything is vanilla at this point.

Example syslog doc:

@timestamp February 28th 2018, 12:16:37.000
_index filebeat-6.2.2-2018.02.28
_score -
_type doc
beat.name es-01
beat.version 6.2.2
fileset.module system
fileset.name syslog
offset 5,651,821
prospector.type log
source /var/log/syslog
system.syslog.hostname XXXXXXXXXXXXXXXXX
system.syslog.message
{"type":"response","@timestamp":"2018-02-28T22:16:37Z","tags":[],"pid":19682,"method":"get","statusCode":200,"req":{"url":"/ui/favicons/favicon-16x16.png","method":"get","headers":{"connection":"upgrade","host":"XXXXXXXXXXXXXXXXX","x-real-ip":"XXXXXXXXXXXXXXXXX","x-forward-for":"XXXXXXXXXXXXXXXXX, XXXXXXXXXXXXXXXXX","x-forward-proto":"http","x-nginx-proxy":"true","accept-encoding":"gzip","cf-ipcountry":"US","x-forwarded-for":"XXXXXXXXXXXXXXXXX","cf-ray":"3f46bfd07ab157b3-IAD","x-forwarded-proto":"https","cf-visitor":"{"scheme":"https"}","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36","accept":"image/webp,image/apng,image/,/*;q=0.8","referer":"https://XXXXXXXXXXXXXXXXX/app/kibana","accept-language":"en-US,en;q=0.9","cf-connecting-ip":"XXXXXXXXXXXXXXXXX"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://XXXXXXXXXXXXXXXXX"},"res":{"statusCode":200,"responseTime":3,"contentLength":9},"message":"GET /ui/favicons/favicon-16x16.png 200 3ms - 9.0B"}
system.syslog.pid 19682
system.syslog.program kibana
system.syslog.timestamp Feb 28 17:16:37

4

The Data looks right

{"type":"response","@timestamp":"2018-02-28T22:16:37Z"

I am not sure what the other data outside of the JSON data is because the first @timestamp at the top is 12:16:37 which is 10 hours different but system.syslog.timestamp Feb 28 17:16:37 but is not part of the json so not sure where that is from. Maybe the formatting from the website is screwed up. you can use the code statement to show it.

Ok yah the system.syslog.timestamp, if you want that to be in sync with the @timestap you need to convert it from a string field to a time/date field. Not sure in filebeat how to do that.

I use logstash and that would just be a simple "date" filter, I am sure you could do it in a pipeline too.

The READ time is @timestamp (UTC) and the log date value was the time it was written to messages at whatever the time zone is

What does your raw log file format look like , in the config doc I see the Timezone in the syslog example but your info provided does not show a timezone in the system.syslog.timestamp

5

When the syslog host doesn't use UTC for its timezone then a conversion is needed. The system module has an option for this called convert_timezone.

You need to delete the existing pipeline for the change to take effect. See: Filebeat 6.2.1 system module does not convert timestamp with timezone

1 Like
6

Still confused so let me ask it a different way - Why do the logs like UFW and Auth that are being monitored from the filebeat.yml showing correctly in Kibana?

I've stood up another fresh box with ES and out of the box syslog is off by 5 hours in Kibana but the syslog itself is correct.

7

Just to follow up. It is working now but I am confused as to why the difference from monitoring via the filebeat.yml and the system.yml

  1. Stop Filebeat
  2. Enable UTC time conversion in system.yml
  3. Delete pipelines
  4. Restart filebeat

Works now.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.