Ingest pipeline filebeat nginx error provided by nginx module - timestamp error UTC+4

Elastic Stack Elasticsearch
1

Versions: Elastic Suite 6.5.1 with Filebeat nginx module.

Thanks to Kibana DevTools, here under is the filebeat-nginx-error-pipeline provided by Filebeat nginx module. Also from here enclosed record, the local time is set to UTC+4h in kibana

GET /_ingest/pipeline/filebeat-6.5.1-nginx-error-pipeline
{
  "filebeat-6.5.1-nginx-error-pipeline" : {
    "description" : "Pipeline for parsing the Nginx error logs",
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            """%{DATA:nginx.error.time} \[%{DATA:nginx.error.level}\] %{NUMBER:nginx.error.pid}#%{NUMBER:nginx.error.tid}: (\*%{NUMBER:nginx.error.connection_id} )?%{GREEDYDATA:nginx.error.message}"""
          ],
          "ignore_missing" : true
        }
      },
      {
        "remove" : {
          "field" : "message"
        }
      },
      {
        "rename" : {
          "field" : "@timestamp",
          "target_field" : "read_timestamp"
        }
      },
      {
        "date" : {
          "formats" : [
            "YYYY/MM/dd H:m:s"
          ],
          "field" : "nginx.error.time",
          "target_field" : "@timestamp"
        }
      },

{
        "_index" : "filebeat-6.5.1-nginxws-2019.07.02",
        "_type" : "doc",
        "_id" : "CzP9smsBWYoED-YAN0ZH",
        "_score" : 5.242066,
        "_source" : {
          "offset" : 12839,
          "nginx" : {
            "error" : {
              "connection_id" : "8136",
              "level" : "warn",
              "pid" : "17507",
              "message" : """upstream server temporarily disabled while reading response header from upstream, client: 192.168.2.4, server: 192.168.2.246, request: "GET /_wss/.ws?v=5&ns=accounts HTTP/1.1", upstream: "http://192.168.2.22:8000/.ws?v=5&ns=accounts", host: "io.xxx.com"""",
              "tid" : "0"
            }
          },
          "prospector" : {
            "type" : "log"
          },
          "read_timestamp" : "2019-07-02T14:01:06.507Z",
          "source" : "/opt/application/nginxws/logs/error.log",
          "fileset" : {
            "module" : "nginx",
            "name" : "error"
          },
          "input" : {
            "type" : "log"
          },
          "@timestamp" : "2019-07-02T16:01:04.000Z",
          "beat" : {
            "hostname" : "i-001c911b-rp-ws-server-15354444231.novalocal",
            "name" : "192.168.2.246",
            "version" : "6.5.1"
          },
          "host" : {
            "name" : "192.168.2.246"
          }
        }
      }
2

I guess the root cause is about the date format in nginx error log file:

2019/07/02 16:01:04 [error] 17507#0: *8136 upstream prematurely closed connection while reading response header from upstream, client: 192.168.2.4, server: 192.168.2.246, request: "GET /_wss/.ws?v=5&ns=accounts HTTP/1.1", upstream: "http://192.168.2.22:8000/.ws?v=5&ns=accounts", host: "io.xtraklif.webcom.orange.com"
2019/07/02 16:01:04 [warn] 17507#0: *8136 upstream server temporarily disabled while reading response header from upstream, client: 192.168.2.4, server: 192.168.2.246, request: "GET /_wss/.ws?v=5&ns=accounts HTTP/1.1", upstream: "http://192.168.2.22:8000/.ws?v=5&ns=accounts", host: "io.xtraklif.webcom.orange.com"
3

See Filebeat Nginx error module @timestamp wrong in Kibana

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.