logstash 5.3.0
filter {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => [
"message", "%{NGINXACCESS} %{GREEDYDATA:message}",
"message", "%{NGINXACCESSAUTH}%{GREEDYDATA:message}",
"message", "%{NGINXERROR}",
"message", "%{PHPLOG}%{GREEDYDATA:message}",
"message", "%{FPMERROR}%{GREEDYDATA:message}",
"message", "%{SYSLOG5424PRI}%{SYSLOGBASE2} %{GREEDYDATA:message}"
]
overwrite => [ "message" ]
}
I am having an issue here where I have a complete parse here for NGINXACCESSAUTH which leaves me with empty result for %{GREEDYDATA:message} and this not rewriting message field to empty, leaving me with messy outcome of message field being the full rsyslog source message as well as all the tags parsed.
program:nginx
logsource:ppdlweb005
nginx_client:10.175.37.27
nginx_auth:-
nginx_time:08/Mar/2018:14:16:24 +0000
nginx_ident:-
nginx_response:200
message:<141>Mar 8 14:16:33 ppdlweb005 nginx 10.175.37.27 - - - [08/Mar/2018:14:16:24 +0000] "HEAD /?_=havemercy11 HTTP/1.1" 200 0 "-" "AppleWebkit/534.1 (KHTML) HbbTV/1.4.1 (+DRM;SureSoft-Browser-3.0;T3;0010;1.0;Manhattan-FVPlay;) FVC/2.0(SureSoft-Browser-3.0;Manhattan-FVPlay;)" SUCCESS 0.001
nginx_bytes:0
http_user_agent:AppleWebkit/534.1 (KHTML) HbbTV/1.4.1 (+DRM;SureSoft-Browser-3.0;T3;0010;1.0;Manhattan-FVPlay;) FVC/2.0(SureSoft-Browser-3.0;Manhattan-FVPlay;) nginx_httpversion:1.1
@timestamp:March 8th 2018, 14:16:33.000
nginx_verb:HEAD
nginx_processing_time:0.001
fvc_role:auth
http_referer:-
fvc_env:staging
syslog5424_pri:141
@version:1
host:ppdlweb005
nginx_ssl_verify:SUCCESS
nginx_request:/?_=havemercy11
timestamp:Mar 8 14:16:33
_id:AWIF-Hov00VaJHdB36R2
_type:logs _index:logstash-2018.03.08
_score: -
Any idea how to go about this apart from removing part of the pattern so there is something for GREEDYDATA to parse?