Apache/nginx logs wrapped in syslog format


I've got a centralized log server that receives apache + nginx logs. I would like to use filebeat on this box to forward these logs into elastic.

The logs are wrapped in the syslog format, e.q:
2019-11-09T11:50:56+00:00 foobar nginx_access: - - [09/Nov/2019:11:50:56 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

so first columns are added by rsyslog therefore i can't use apache/nginx filebeat module

Obviously I could change rsyslog format to exclude these 3 columns (ts, host, tag) but wondering if it's possible to pre-process on the filebeat level so I can use filebeat modules rather than writting my post processors


You can use a processor to split your log into fields before to put it in the Filebeat index, the grok processor fits to what you want to make
there are also more processors that you can check if you think that grok is not the one you need https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-processors.html

Hi @w_o_j_t_e_k,

it should be possible to override the module's input settings and add a dissect or script processor in front that discards the prefix. The original configuration applied by the nginx access log module seems to apply the add_locale processor.

Let us know if that works for you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.