NGINX logs - Provided Grok expressions do not match field value

Elastic Stack Beats
1

I'm trying to write a grok filter for my nginx log.

the filter:

filter {
  if [fileset][module] == "nginx" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => '%{IPV4} - - \[%{HTTPDATE:timestamp}\]\" "%{WORD:method} %{URIPATH:uri_path}(?:%{URIPARAM:uri_params})? HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}" "%{DATA:clientip}" "%{DATA:session}" "%{NUMBER:responsetime}" %{DATA:domain}' }

      }
      mutate {
        convert => ["response", "integer"]
        convert => ["bytes", "integer"]
        convert => ["responsetime", "float"]
      }
    }
  }
}

it works, when I test it with the stdin input.

But when I use the beats input on port 5044, I see strange things in kibana:

Provided Grok expressions do not match field value: [192.168.32.98......

How to debug what is the issue ?

Here a sample line from my nginx logs:

192.168.32.98 - - [18/Apr/2018:11:36:51 +0200]" "GET /?ping HTTP/1.1" 200 63718 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" "194.76.219.19" "3294aeff1d3031a4c880ac7497688473" "0.105" "www.example.com"
2

Which version of NGINX are you using? Do you have special log_format option in your NGINX config?

3

Yes, I have a custom log format, that's why I'm using my custom pattern matching in the pasted grok config.

As I already have written: it works when I testing it with the stdin plugin, It works as well in the grok debugger.

Only when using the beats input, I got parsing errors visible in kibana.

The log format:

log_format  main  '$remote_addr - $remote_user [$time_local]" "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$cookie_client_id" "$request_time" "$host"';
4

Are you using the NGINX Filebeat module? If yes, it's possible that the error message is coming from the module. Messages coming from Filebeat go through the default modules pipeline. That's not appropriate in your case.

I would customize the existing NGINX pipeline to match your log format. Load it to Elasticsearch with a new ID. Then set the option pipeline of the input to use it.

- type: log
  enabled: true
  paths:
    - {{ path/to/your/log }}
  pipeline: {{ custom-pipeline-id }}

Thus, logs coming from this input would go through the customized pipeline.

5

I don't get it. And please bear with me, I read all the docs, but feel quite lost.

I just want to pipe my logs from nginx through logstash to ES and be able to visualise in a kibana dashboard.

Should I abandon the usage of filebeat in my case?

Or what are the steps to achieve my goal?

6

I got it working now.

I removed the filebeat nginx module, and I'm using filebeat only, now it works flawlessly.

Is there a way to use the filebeat nginx module with a custom nginx log format?

7

I am glad you got it working.

Filebeat modules are made for smoothing the getting started experience of new users. So by default its pipeline does not support custom log formats. However, advanced Filebeat users can modify the patterns in module/nginx/access/ingest/default.json and their own format. After the pipeline is updated, it needs to be loaded to the Ingest node. Right now in order to update an existing pipeline, you need to delete it manually and then load the new version. But in the next release filebeat.update_pipelines is introduced. If it's set to true pipelines are always updated.

2 Likes
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.