Hi @BDeveloper
I have and update / solution for you. You may need to adjust etc... I picked some names for fields etc, there is no guarantee that every log line will parse / work.
We will start from where you left off with the Quickstart setting up the Filebeat and nginx module.
Then The Macro Steps are
- Clone the Existing nginx Access Module Pipeline to become our custom pipeline
- Add the new Grok Pattern Etc to support your custom format
- Set the the modules to use our new custom pipeline
- Run filebeat
Here are the files
In the next Post (perhaps tomorrow) I will show you how to quickly build / test ingest pipeline / groks etc.
These are the steps... .follow them very very closely. I am doing them through the UI but of course in reality I do it all through the API. with the links I put above
Clone The Existing Ingest Pipeline, Name It
Add the new Grok Pattern and
IMPORTANT Move it to the top so that it matches first (perhaps more on that later) Be Careful with the cut and past.
Save the Grok
(%{NGINX_HOST} )?"?(?:%{NGINX_ADDRESS_LIST:nginx.access.remote_ip_list}|%{NOTSPACE:source.address}) - (-|%{DATA:user.name}) \[%{HTTPDATE:nginx.access.time}\] "%{DATA:nginx.access.info}" %{NUMBER:http.response.status_code:long} %{NUMBER:http.response.body.bytes:long} "(-|%{DATA:http.request.referrer})" "(-|%{DATA:user_agent.original})" "-" "(-|%{IPORHOST:nginx.access.host.name})" sn="(-|%{DATA:nginx.access.host.domain})" rt=(-|%{NUMBER:nginx.access.request_time:float}) ua="(-|%{DATA:nginx.access.upstream_addr})" us="(-|%{DATA:nginx.access.upstream_status})" ut="(-|%{NUMBER:nginx.access.upstream_response_time:float})" ul="(-|%{NUMBER:nginx.access.upstream_response_length:long})" cs=-
IMPORTANT Save the Pipeline
Now Modify the nginx.yml to use the new custom pipeline.
- module: nginx
# Access logs
access:
enabled: true
# Set the custom pipeline
input.pipeline: filebeat-8.11.3-nginx-access-pipeline-custom
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: ["/Users/sbrown/workspace/sample-data/discuss/discuss-custom-nginx.log"]
Start filebeat
Check Discover
