Hi gang,
Completely out of my wits on this one...
Here's a log line (NGINX custom format):
13.88.158.6 - - [14/Mar/2018:00:02:03 +0000] "GET /issue/wsfed?wa=wsignout1.0 HTTP/1.0" 499 0 "https://msft2017r2-wus2-prd.whatever.com/app.aspx" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299" "-" "pass.whatever.com" sn="passproxy-prod.whatever.com" rt=49.171 ua="13.88.158.6:443" us="-" ut="-" ul="0" cs=-Netherlands NL NH
It can be parsed this way with grok:
filter {
grok {
match => [ "message","%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
overwrite => [ "message" ]
}
but the result is not accurate.
Here's an expression that will fly in https://grokdebug.herokuapp.com and other grok debuggers:
%{IPORHOST:remote_addr} %{USER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] \"%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" %{INT:status} %{NUMBER:bytes_sent} \"%{DATA:http_referer}" \"%{DATA:http_user_agent}" \"%{DATA:http_x_forwarded_for}\" \"%{DATA:host}\" sn=\"%{DATA:server_name}\" rt=%{DATA:request_time} ua=\"%{DATA:upstream_addr}\" us=\"%{DATA:upstream_status}\" ut=\"%{DATA:upstream_response_time}\" ul=\"%{DATA:upstream_response_length}\" %{GREEDYDATA:extra_fields}
... but that gets me a _grokparsefailure in logstash.
Any idea how I shall build the "match" statement in logstash.conf so that my (legal) expression does not result in a _grokparsefailure ?
I've tried everything, working my way up from what is working. Every time I try to add something here I get this dreaded grokparsefailure:
match => [ "message","%{COMBINEDAPACHELOG}[ADD-SOMETHING-HERE]+%{GREEDYDATA:extra_fields}"]
Thanks much,
Fred