Filebeat Nginx error module @timestamp wrong in Kibana

Elastic Stack Beats
1

Hello

I'm having an issue with filebeat 6.2.4 with the Nginx error module. The access module works fine and the data pulls into kibana with the correct time. However, the error logs show up 4 hours late with the default config. I edited the pipeline config to EST but, now the logs come in 1 hour early (in the future).

Is there a list of timezones supported since EDT didn't seem to work? Or am I going about this all wrong?

From error pipeline:

"date": {
  "field": "nginx.error.time",
  "target_field": "@timestamp",
  "timezone": "EST",
  "formats": ["YYYY/MM/dd H:m:s"]
}

From access pipeline:

"date": {
  "field": "nginx.access.time",
  "target_field": "@timestamp",
  "formats": ["dd/MMM/YYYY:H:m:s Z"]
}

Access logs:
104.63.177.249 - - [04/Jun/2018:12:09:56 -0400] "GET

Error logs:
2018/06/04 12:03:06 [error] 16906#0: *1700526 open()

Logs go directly to ES

output.elasticsearch:
hosts: ["elk.n2ntech.com:9200"]
index: "filebeat-%{[beat.version]}-%{+yyyy.MM}"

Let me know what you think please. I've searched around and found similar issues but, I haven't found a good fix. Especially since it looks like it should work by default with this

Thank you!

EDIT: Received a PM from @Lucas_Pereira_Borges that helped!

"Hello,

I've the same problem and discovered a fix for that.
Update your nginx-error-pipeline in the date section add the timestamp"

"date": {

"field": "nginx.error.time",

"target_field": "@timestamp",

 "formats": [

  "YYYY/MM/dd H:m:s"

],

"timezone": "America/Sao_Paulo"

}
2

Upon enabling more modules (system) for this system it appears that only nginx access is showing up as EST in Kibana while everything else uses UTC. Is there a way to have the access logs follow UTC?

Acccess
@Timestamp - June 4th 2018, 13:31:59.000
read_timestamp - 2018-06-04T 17:31:59.948Z

Error
@Timestamp - June 4th 2018, 09:33:25.000
read_timestamp - 2018-06-04T 17:33:28.674Z

3

I also tried to find a list of the supported format but didn't find one. Not sure if this list here helps (http://joda-time.sourceforge.net/timezones.html). Can you try one of these?

I think the problem here is that the access log contains the time zone in each timestamp and explains the pattern that is a little bit different. For processing it would be best the error log would do the same but it does not seem to be the case. Do you know if you can configure nginx to log in UTC time instead of trying to do the conversion?

4

Thank you for your response! I'll give those a shot as soon as I'm back to that cluster (Friday). There's no remote access to it

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.