Nginx access log using grok filter

Elastic Stack Logstash
1

My nginx access log format as below, there certain access log without the
"$request_time" "$http_x_forwarded_for" $http_host ' field, therefore, for certain access log, the grok filter not working, is there anyway to automatic fill the field with - so that other field of the message will be process?

 log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$request_time"  "$http_x_forwarded_for" $http_host ';


 grok {
        match => { "message" => "%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method}%{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{NUMBER:request_time}\"  \"%{DATA:http_x_forwarded_for}\" %{IPORHOST:http_host} " }
        remove_field => "message"
       
    }
1 Like
2

You could have a catch all %{GREEDYDATA:payload} at the end of the grok options.

Something like
match => { "message" => ["%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] \"%{WORD:method}%{DATA:url} HTTP/%{NUMBER:http_version}\" %{NUMBER:response_code} %{NUMBER:body_sent:bytes} \"%{DATA:referrer}\" \"%{DATA:agent}\" \"%{NUMBER:request_time}\" \"%{DATA:http_x_forwarded_for}\" %{IPORHOST:http_host} ", %{GREEDYDATA:payload} ] }

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.