Grok tests pass, logstash still borks with "_grokparsefailure"

Elastic Stack Logstash
1

Hello all,

working on parsing firewall logs and I have it working at http://grokconstructor.appspot.com/
When I put the same pattern in Logstash, it comes back with a lovely "_grokparsefailure"

log line:

2015-07-08T09:21:12-05:00 SWRG-PA200 1,2015/07/08 09:21:11,001606020851,TRAFFIC,end,1,2015/07/08 09:21:11,10.5.2.1,65.55.7.141,66.196.247.82,65.55.7.141,Any-To-Outside,,,ssl,vsys1,Inside-Wired,Outside,ethernet1/2,ethernet1/4,swrg-raspi,2015/07/08 09:21:11,64048,1,62238,443,11943,443,0x400053,tcp,allow,6048,2522,3526,17,2015/07/08 09:20:56,1,any,0,157064842,0x0,10.0.0.0-10.255.255.255,US,0,10,7,tcp-fin

grok filter line:

match => { "message" => "%{TIMESTAMP_ISO8601}%{SPACE}%{HOSTNAME:host}(\s*(\:)?(\-)?\s)%{INT}%{SPACE},%{TIMEGEN},%{INT},%{WORD:PAType},%{WORD:SubType},%{INT},%{TIMEGEN},%{IP:sourceIP},%{IP:destIP},%{IP:NATSrcIP},%{IP:NATDstIP},%{HYPHENWORD:Rule}"}

The custom definitions are as follows:

TIMEGEN (%{YEAR}/%{MONTHNUM2}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND})
HYPHENWORD (\w+(?:[-\s]\w+)+|(\w*))(\(?\d+\))?

The grok passes every time with grokconstructor, yet fails in Logstash.

Any ideas?

Thanks!

1 Like
2

Your pattern worked fine for me (except I replaced occurrences of TIMEGEN and HYPHENWORD with their definitions to avoid having to deal with custom pattern files). Do you use a syslog input? I ask because it uses the grok library internally and will add _grokparsefailure tags if the syslog data doesn't match the expectations.

3

All the logs in question are coming from another server running logstash-forwarder which is getting them via rsyslog.

If I remove the %{HYPHENWORD} from the end, it will pass the filter.

I'll attempt to put the regex definition and see if that works.

4

Found the problem.
In the pattern file, I had a tab between the term and the definition.
I believe it's supposed to be just a space between them.
Now that the file reads like this:

HYPHENWORD (\w+(?:[-\s]\w+)+|(\w*))((?\d+))?

And not:

HYPHENWORD(tab)(\w+(?:[-\s]\w+)+|(\w*))((?\d+))?

It's working.

5

Ouch. That sounds like a bug to me. Grok didn't complain about the pattern file being malformed?

6

Not that I saw, however, I may have missed it in with all the stuff that --debug generates.
Certainly no mention in logstash.err

7

I'd recommend you raise a GH issues on this :slight_smile: