Grok from nginx-module gets error

Elastic Stack Beats
1

Hello!
I have nginx-module enabled in filebeat.yml:

filebeat.modules:
- module: nginx
  access:
    var.paths: ["/testvar/nginx/access.log"]
  error:
    var.paths: ["/testvar/nginx/error.log"]

output.elasticsearch:
  hosts: ["localhost:9200"]

setup.kibana:
  host: "localhost:5601"

And all logs like:

127.0.0.1 - 127.0.0.2 - [02/Aug/2018:11:57:45 +0300] "adress.ru" "POST /smth"

are parsed in ElasticSearch. But with logs like:

-- 127.0.0.2 - [02/Aug/2018:11:57:45 +0300] "adress.ru" "POST /smth"

Kibana gets filed message.error with entry like:

Provided Grok expressions do not match field value:[...]

Any thoughts what I can do to improve my nginx-module?

2

Thanks @g.myznikov.tinkoff

I have added the above line in our integration test suite and I can reproduce the error:

AssertionError: not error expected but got: {u'beat': {u'hostname': u'sashimi', u'name': u'sashimi', u'version': u'7.0.0-alpha1'}, u'@timestamp': u'2018-08-08T14:20:02.157Z', u'offset': 2346, u'fileset': {u'name': u'access', u'module': u'nginx'}, u'source': u'/Users/ph/go/src/github.com/elastic/beats/filebeat/module/nginx/access/test/access.log', u'host': {u'name': u'sashimi'}, u'error': {u'message': u'Provided Grok expressions do not match field value: [-- 127.0.0.2 - [02/Aug/2018:11:57:45 +0300] \\"adress.ru\\" \\"POST /smth\\"]'}, u'input': {u'type': u'log'}, u'message': u'-- 127.0.0.2 - [02/Aug/2018:11:57:45 +0300] "adress.ru" "POST /smth"', u'prospector': {u'type': u'log'}}

It look like that the following grok pattern is stricter than some real world use case.

 "pattern_definitions": {
        "IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
},

Can you tell me what is the version of nginx you are running? Also are you using the default log format for nginx or a custom one?

3

I use nginx version 1.12.1. I have not changed log format so it is probably default.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.