Nginx access and error logs show incorroct value in kibana dashboard

[stephenb (Stephen Brown)]

Ok here we go

First your error logs seem to parse fine so I think you are good there.

I am pretty sure the other number is request body bytes so I created a custom pipeline.

So here is what you can do....

1. I created a custom pipeline it was a pretty simple fix. You can find it here bvader that is me.
   Be very careful copying make sure you copy from the raw.

2. Run those commands in the dev tool that will add the custom pipeline...

   DELETE _ingest/pipeline/filebeat-7.12.0-nginx-custom-access-pipeline

   PUT _ingest/pipeline/filebeat-7.12.0-nginx-custom-access-pipeline
   {
   ...
   }

   GET _ingest/pipeline/filebeat-7.12.0-nginx-custom-access-pipeline

If you want to know the difference it is the grok.

default

... %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} \"(-|%{DATA:http.request.referrer})\" ...

The new one with the new field.

....%{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} %{NUMBER:http.request.body.bytes:long} \"(-|%{DATA:http.request.referrer})\"....

3. Now this is important and it can be confusing do not put the new pipeline on the output, when you configure it on the input it will end up in the right metadata fields and will override the default pipeline which is what we want. If you put it on the output the default pipeline will override the custom pipeline we just created and it will not work.

filebeat/modules.d/nginx.yml

- module: nginx
  # Access logs
  access:
    enabled: true
    input:
      pipeline: filebeat-7.12.0-nginx-custom-access-pipeline
    var.paths: ["/var/log/nginx/access.log"]

4. Clean up the filebeat registry from the filebeat directory
   rm -fr ./data

5. Run Filebeat

Here is a little screen shot of the 3 rows I even put in a real IP address. The user agent failed I think because they are fake.

The Errors worked too...

[fati]

Thank you so much @stephenb

This is my nginx log config:

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent $bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$request_body"';

First I change log format to standard and check the result, if no problem I will try your solution and give feedback.

[fati]

Thank you so so much @stephenb
I changed my nginx log config based on standard format:

'$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent $bytes_sent "$http_referer" '
                      '"$http_user_agent"  ';

And now everything is ok
I will try your solution in custom log format and will feedback.

Thanks again.