Apache module bad timestamp from error log file

[Aurelien_Bras (Aurélien Bras)]

Hi here,

Using lastest filebeat/elasticsearch/kibana (6.4.1), when sending apache error log, it read timestamp writted in local time from log file and override timestamp in UTC format (sended by filebeat), then all logs is seen by Kibana 2 hours in the future (because my timezone is GMT+2).

The timestamp in the filebeat event is correct, but it will appear on kibana at 22h40, with bad timestamp calculated from log file.

The filebeat event :

2018-09-25T20:40:08.753+0200	DEBUG	[publish]	pipeline/processor.go:308	Publish event: {
  "@timestamp": "2018-09-25T18:40:08.753Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.4.1",
    "pipeline": "filebeat-6.4.1-apache2-error-pipeline"
  },
  "beat": {
    "version": "6.4.1",
    "timezone": "+02:00",
    "name": "zef-web1.intra",
    "hostname": "zef-web1.intra"
  },
  "host": {
    "name": "zef-web1.intra"
  },
  "source": "/var/log/httpd24/error_log",
  "offset": 10805,
  "message": "[Tue Sep 25 20:40:08.250210 2018] [core:info] [pid 27615:tid 140099626960640] [client 100.64.2.243:38462] AH00128: File does not exist: /home/tools/robots.txt",
  "fileset": {
    "name": "error",
    "module": "apache2"
  },
  "prospector": {
    "type": "log"
  },
  "input": {
    "type": "log"
  }
}

The log file :

 [Tue Sep 25 20:40:08.250210 2018] [core:info] [pid 27615:tid 140099626960640] [client 100.64.2.243:38462] AH00128: File does not exist: /home/tools/robots.txt

The part of the pipeline with the bad code :

  {
    "date": {
      "formats": [
        "EEE MMM dd H:m:s YYYY",
        "EEE MMM dd H:m:s.SSSSSS YYYY"
      ],
      "ignore_failure": true,
      "field": "apache2.error.timestamp",
      "target_field": "@timestamp"
    }

Removing the code above directly in the pipeline remove the issue.

[ruflin]

What you would need in your case is to set the timezone in the ingest processor: https://www.elastic.co/guide/en/elasticsearch/reference/master/date-processor.html Otherwise everything is converted to UTC.