Apache/nginx logs wrapped in syslog format

Elastic Observability Logs
1

Hi,

I've got a centralized log server that receives apache + nginx logs. I would like to use filebeat on this box to forward these logs into elastic.

The logs are wrapped in the syslog format, e.q:
2019-11-09T11:50:56+00:00 foobar nginx_access: 202.18.3.162 - - [09/Nov/2019:11:50:56 +0000] "GET / HTTP/1.1" 301 178 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"

so first columns are added by rsyslog therefore i can't use apache/nginx filebeat module

Obviously I could change rsyslog format to exclude these 3 columns (ts, host, tag) but wondering if it's possible to pre-process on the filebeat level so I can use filebeat modules rather than writting my post processors

Regards

2

You can use a processor to split your log into fields before to put it in the Filebeat index, the grok processor fits to what you want to make
https://www.elastic.co/guide/en/elasticsearch/reference/current/grok-processor.html
there are also more processors that you can check if you think that grok is not the one you need https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-processors.html

3

Hi @w_o_j_t_e_k,

it should be possible to override the module's input settings and add a dissect or script processor in front that discards the prefix. The original configuration applied by the nginx access log module seems to apply the add_locale processor.

Let us know if that works for you.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.