I'm using the below pattern to match these example logs:
2018-11-12 10:21:33,596[event-thread-1]INFO (HTTP-ENTRY)-ef8684ae-72a4-4ad7-94f9-603e7999b3a4 Get Request to /session with method POST from 10.134.236.205, 169.193.66.107 to 169.193.125.49:8008 return StatusCode 200 process in 34 millionSec client is myclient
2018-11-12 10:21:33,596[event-thread-1]INFO (HTTP-ENTRY)-ef8684ae-72a4-4ad7-94f9-603e7999b3a4 Get Request to /session with method POST from 10.111.236.205 to 169.123.234.49:8008 return StatusCode 200 process in 34 millionSec client is myclient
%{DATESTAMP:timestamp}\[%{DATA:thread}\]%{LOGLEVEL:logLevel}\s*\(%{DATA:program}\)\-%{UUID:id}\s*%{GREEDYDATA}Get Request to\s*%{URIPATHPARAM:request}\s*%{GREEDYDATA}with method\s*%{WORD:method}\s*%{GREEDYDATA}from\s*(?:(?:%{HOSTPORT:client_ip}|%{IP:client_ip})|((?:%{HOSTPORT:forwarding_ip}|%{IP:forwarding_ip}),\s*(?:%{HOSTPORT:client_ip}|%{IP:client_ip})))\s*to\s*(?:%{HOSTPORT:server_ip}|%{IP:server_ip})\s*%{GREEDYDATA}return StatusCode\s*%{NUMBER:statuscode}\s*%{GREEDYDATA}process in\s*%{NUMBER:duration}(?:%{GREEDYDATA}\s*is\s*%{USERNAME:client})?
In http://grokconstructor.appspot.com/do/match there is absolutely no problem.
However when there is a "forwarding ip" included in the logs (log line example #1, comma separated IPs) for some reason logstash puts a comma, space then a hyphen after SERVER_IP... E.G. -> "169.123.234.49:8008, - "
I have keep_empty_captures set to "true".
I cannot explain this behaviour. Any help?
EDIT - checking more logs coming in, it seems to be adding ", -" to IP fields everywhere. Is this something to do with keep_empty_captures maybe?