_grokparsefailure when parsing IIS logs if the %{URIPATH:page} has square brackets [] in it

Elastic Stack Logstash
1

Hi,

3rd day after I installed ELK - so I am still very new at all this.

I have noticed that some of my IIS log files have [] (square brackets?) in the %{URIPATH:page} and this is causing a _grokparsefailure.

My GROK is:

grok {
    match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostip} %{WORD:method} %{URIPATH:page} %{NOTSPACE:query} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientip} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:timetaken}"]
	}
  
  #Set the Event Timestamp from the log
	date {
		match => [ "timestamp", "YYYY-MM-dd HH:mm:ss" ]
		target => "@timestamp"
		timezone => "Europe/London"
    }

A sample IIS log entry that fails to parse is:

2015-10-03 12:39:29 10.2.10.11 GET /Pictures/7751861-IMG_4866[1]_thumbnail.PNG 5343 443 704742 192.168.1.1 Mozilla/5.0+(Windows+NT+10.0;+WOW64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/45.0.2454.99+Safari/537.36 https://www.mywebsite.com/Home/PicturesPanel 200 0 0 125

Can GROK change the square brackets to round brackets as it parses each line?

If this is possible please would you be kind enough to point me in the right direction.

As always, any help offered will be very much appreciated.

Martin

2

Can GROK change the square brackets to round brackets as it parses each line?

No, grok doesn't do replacements. Besides, why would you want to change the source data like that?

Replace the use of URIPATH with NOTSPACE. All spaces in URI paths should be encoded as + or %20 so it should be safe to use a space as a delimiter to the subsequent token.

1 Like
3

Once again a massive 'thank you' to @magnusbaeck as your suggestion worked perfectly!

I now need to work out the best way of getting the millions of failed lines to re-try without creating duplicates of those that worked the first time.

I guess worst case I can delete all indexes and restart the import from scratch - but that is a job for Monday.

Cheers
Martin