Filebeat assumes UTC?

[beatsme]

I'm using filebeat to import syslog messages. When looking at the ES document it appears filebeat incorrectly assumes UTC:

ES document:

"@timestamp": "2017-04-01T15:26:51.000Z"

Syslog message

"timestamp": "Apr 1 15:26:51"

OS:

Sat Apr 1 15:26:51 CEST 2017

Should filebeat not store the @timestamp as 13:26:51?

[andrewkroh]

Could you explain your setup more. Are you using Logstash or Ingest node to parse logs? What does the config look like?

[beatsme]

Hi Andrew,

I'm not using Logstash or Ingest, I'm directly sending syslog from filebeat to ES. My filebeat config:

filebeat.modules:
- module: system
  syslog:
    enabled: true
    var.paths: ["/var/log/auth.log", "and more logs..."]

- module: apache2
  access:
    enabled: true
    var.paths: ["/var/log/apache.log"]

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

Thanks!

[beatsme]

Forgot to mention that I'm on ES/Kibana/filebeat 5.3.0 from the ES repo. The user-agent/geoip plugins are enabled in ES.

[andrewkroh]

You are using the system module in Filebeat. Under the hood this is using Elasticsearch's Ingest Node feature to parse the syslog logs. It then overwrites the @timestamp field added by Filebeat (which is actually in UTC) with the interpreted timestamp from the log line. Syslog doesn't include the timezone in the logs that it writes and the Ingest Node date processor assumes UTC.

I would recommend running all your systems with UTC time. But you can modify the pipeline to use a custom timezone (note that f you have multiple systems with different timezones sending syslogs this won't work). Wherever you installed Filebeat, there is a module/system/syslog/ingest/pipeline.json file. You can add a timezone to it. For example:

    {
      "date": {
        "field": "system.syslog.timestamp",
        "target_field": "@timestamp",
        "timezone": "EST",
        "formats": [
          "MMM  d HH:mm:ss",
          "MMM dd HH:mm:ss"
        ],
        "ignore_failure": true
      }
    }

Save the change. Stop Filebeat. Delete the pipeline from Elasticsearch. And restart Filebeat.

You can delete the pipeline with

curl -XDELETE http://elasticsearch:9200/_ingest/pipeline/filebeat-5.3.0-system-syslog-pipeline

[beatsme]

Thanks a lot Andrew, understand how that works now. Is it possible to configure the custom pipeline I would need for my use case? I'd like to avoid losing my modification when I upgrade filebeat. It does not appear to be a 'var' of the module.

(agree that running the OS in UTC would be better...)

[andrewkroh]

At the moment (AFAIK) there isn't way to do so. Probably @tudor has given this some thought already. I can ask him next week.

[tudor (Tudor Golubenco)]

Thanks for the feedback, I think a variable for the timezone makes sense in this module. Could you open a Github ticket for it, please? It's not trivial to implement because currently we insert the pipelines "as is" without expanding variables into them, but I think that's something we can add.