How to parse filebeat access log with vhost?

[nycterent]

hello,

is there a simple way to parse virtual host from nginx logs with filebeat? any hints would be appreciated :)]

[ruflin]

Are you using the current nginx module? Which version?

Can you share an example log line?

[nycterent]

I am using nginx module, my filebeat version is 6.2.4.

I have successfully modified grok pattern to work with such settings in nginx log:

log_format main '$host $remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"' ' $request_time ' '$upstream_response_time';

grok pattern:
"patterns": [
"""%{IPORHOST:nginx.access.http_host} ?%{IP_LIST:nginx.access.remote_ip_list} - %{DATA:nginx.access.user_name} [%{HTTPDATE:nginx.access.time}] "%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP/%{NUMBER:nginx.access.http_version}" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} "%{DATA:nginx.access.referrer}" "%{DATA:nginx.access.agent}" %{NUMBER:nginx.access.request_time:float} %{NUMBER:nginx.access.upstream_time:float}"""
],

log example:

example.com 127.0.0.1 - - [25/May/2018:11:02:40 +0000] "GET / HTTP/1.1" 200 14 "-" "-" "-" 0.013 0.012

only visualisations does not take these new fields into account, but I can live without it.

[ruflin]

I'm glad you found a solution that works for you.

The tricky part about the nginx log is that users can configure the format themself which makes it tricky to have a grok pattern that matches all of them.