Grokparsefailure migrating from 1.4.1 to 2.3.1

Elastic Stack Logstash
#1

I'm migrating from logstash 1.4.1 to 2.3.1 and grok is giving me troubles, seems like it's not behaving like it used to. Is giving me grokparsefailure on this data:

Apr 18 09:25:20 10.0.0.10 date=2016-04-18 time=09:19:10 devname=DEV-NAME-01 devid=885700C78644 logid=0000000009 type=traffic subtype=forward level=notice vd=root srcip=10.0.0.15 srcport=58342 srcintf="INTERNAL" dstip=192.51.16.12 dstport=443 dstintf="DEST_01" sessionid=1577369893 poluuid=a6cc157c-abe1-51e4-1d9c-0e42462e2584 dstcountry="Germany" srccountry="Reserved" service=HTTPS wanoptapptype=web-cache proto=0 duration=0 policyid=1 wanin=0 rcvdbyte=0 wanout=0 lanin=187 sentbyte=187 lanout=72 utmaction=block countweb=1 crscore=30 craction=8

Using this simple testing pattern:

match => { "message" => "%{MONTH:month} %{GREEDYDATA}" }

If I just

match => { "message" => "%{GREEDYDATA}" }

It doesn't show any problems so I know it is processing the message.

What I'm I missing here? This is just a conf that is working on a currently running 1.4.1.

#2

Found the problem, the grok pattern for MONTH doesn't like when the first letter of the month is lowercase. I modified the pattern in ./vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.5/patterns/grok-patterns to be case insensitive and now it works.

#3

That'd make a great PR :slight_smile:

#4

Found the problem, the grok pattern for MONTH doesn't like when the first letter of the month is lowercase.

But in your example string the month starts with a capital letter...?

Be careful if your logs aren't actually in English.

#5

Yes, the original logs come with the first letter uppercase, then I lowercase the whole event and then filter it with grok.

That's why it was failing, I found out about it after I commented out the part that "lowercases" the message and the problem stopped. Then I checked the grok patterns that come with logstash and saw that the MONTH one was coded so that the first letter had to be uppercase. I thought to myself that the best way to fix it was to make it case insensitive and avoid future problems, I just did that and it started working.

#6