How to debug "Grok regexp threw exception"

Elastic Stack Logstash
#1

Hi, i have many grok rules and recently i notice this in the logs:

{:timestamp=>"2016-04-14T16:19:46.989000+0200", :message=>"Grok regexp threw exception", :exception=>"", :level=>:warn}

i have no exception text, so i have no idea how to debug this...

It would be helpful if logstash tell us what grok it was using... or at least the file where it is

Thanks in advance for the help
Best regards
Daniel

#2

Normally when I get a grok exception error, the tags field is populated on the record that failed. We don't use that field for anything so when it exists I know something went wrong.

"tags": [
      "_grokparsefailure"]

Assuming you don't normally use the tags field, you can find any instance of it existing using "exists:tags" in Kibana.
If your Grok rules create new fields, then just look at what fields don't exists but should have, and then you know which rule went wrong.
Once you have the record you can use http://grokconstructor.appspot.com/do/match to help debug where it went wrong.

#3

From what i found it is related with groks with regexp ranges ([]) that include the ] (and maybe other characters).
In my config, a valid regexp [^]] was giving problems in logstash.... I had to replace that with [^]] and after that the warning disappeared
I think i had another example, but from what i remember, it was also a problem with the regexp ranges

For other users searching for this:
no grok test tool could point me in the correct direction, nor logstash configtest, only by trial and error i could found the issue. I started to include a tag in every filter file, to see what filters where applied to each message and checked all failed _grokparsefailure messages to understand why did they failed. After some cleanup, i finally managed to find a example that the https://grokdebug.herokuapp.com/ and https://grokconstructor.appspot.com/ could parse without problems, but logstash kept rejecting. Then started to break that grok (removing blocks and resent the message) until i found the block that was failing... another fight was to find how to fix it, but luckily that was faster than the previous steps :slight_smile:

Maybe elastic can check for this special case in next versions on the config test

#4