Help parsing custom nginx logs using Filebeat and Ingest Pipelines

stephenb · December 28, 2023, 2:19am

I have and update / solution for you. You may need to adjust etc... I picked some names for fields etc, there is no guarantee that every log line will parse / work.

We will start from where you left off with the Quickstart setting up the Filebeat and nginx module.

Then The Macro Steps are

Clone the Existing nginx Access Module Pipeline to become our custom pipeline
Add the new Grok Pattern Etc to support your custom format
Set the the modules to use our new custom pipeline
Run filebeat

Here are the files

The Pipeline Custom Pipeline

The Log File

The nginx.yml

In the next Post (perhaps tomorrow) I will show you how to quickly build / test ingest pipeline / groks etc.

These are the steps... .follow them very very closely. I am doing them through the UI but of course in reality I do it all through the API. with the links I put above

Clone The Existing Ingest Pipeline, Name It

Add the new Grok Pattern and
IMPORTANT Move it to the top so that it matches first (perhaps more on that later) Be Careful with the cut and past.
Save the Grok

(%{NGINX_HOST} )?"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address}) - (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] "%{DATA:nginx.access.info}" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} "(-|%{DATA:http.request.referrer})" "(-|%{DATA:user_agent.original})" "-" "(-|%{IPORHOST:nginx.access.host.name})" sn="(-|%{DATA:nginx.access.host.domain})" rt=(-|%{NUMBER:nginx.access.request_time:float}) ua="(-|%{DATA:nginx.access.upstream_addr})" us="(-|%{DATA:nginx.access.upstream_status})" ut="(-|%{NUMBER:nginx.access.upstream_response_time:float})" ul="(-|%{NUMBER:nginx.access.upstream_response_length:long})" cs=-

IMPORTANT Save the Pipeline

Now Modify the nginx.yml to use the new custom pipeline.

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set the custom pipeline
    input.pipeline: filebeat-8.11.3-nginx-access-pipeline-custom

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/Users/sbrown/workspace/sample-data/discuss/discuss-custom-nginx.log"]

Start filebeat

Check Discover