Ingest custom nginx log format

Elastic Stack Beats
1

I'm using filebeat, my setup is pretty plain: I index nginx log files. Now I wan't to log just one more field ($http_host). Let's say I prefix every line with that:

access_log /var/log/nginx/access.log main

becomes

log_format host_main  '$http_host $remote_addr - $remote_user [$time_local] "$request" '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for"';
...
access_log /var/log/nginx/access.log host_main;

I'd assume that I can configure nginx module to interprete log lines using a different pattern. I cannot find anything in the filebeat configuration hinting to that.
How do I add the host_name to my log entries?

2

Unfortunately, the nginx module of Filebeat does not support by default to add more patterns.
But to work around the issue you could install your own nginx pipeline on Elasticsearch.
The pipeline which is loaded to ES is located under module/nginx/acces/ingest/default.json. You could add one more pattern to the first grok processor of the pipeline to parse your messages correctly. After you add the extra pattern, you need to upload the pipeline to ES again using /filebeat setup --pipelines -modules=nginx.
Let me know if you need further help with that.

3 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.