_grokparsefailure also after Grok Constructor

Elastic Stack Logstash
1

HI, I have a challenge that I have been trying to resolve now for some time however not making positive progress. I have logs that I would like to filter and did Incremental Grok Construct and all is matching, however when running this into Elaticsearch I get the _grokparsefailure error.

Here is a sample log:
Jul 22 03:30:10 192.29.58.44 400 <11>1 2016-07-22T03:17:29+01:00 10.31.12.12 1 - - - 2016-07-22T03:29:14.649+01:00 ABC-0000-WWWW03 RT_IDS - RT_SCREEN_TCP_LS [junos@1111.1.1.1.1.88 logical-system-name="TEST-INTERNET" attack-name="TCP sweep!" source-address="58.18.186.131" source-port="45633" destination-address="106.225.87.224" destination-port="22" source-zone-name="INTERNET-TEST" interface-name="xe-1/0/13.9" action="drop"]

The filter I am using is:
grok {
match => ["messages", "%{SYSLOGTIMESTAMP:Date1} %{SYSLOGHOST} %{WORD:Nr} %{SYSLOGPROG} %{TIMESTAMP_ISO8601:Date2} %{IP} %{WORD} - - - %{TIMESTAMP_ISO8601:Date} %{HOSTNAME} RT_IDS - RT_SCREEN_TCP_LS %{SYSLOG5424PRINTASCII} %{SYSLOG5424PRINTASCII} %{GREEDYDATA}"]
tag_on_failure => []
add_tag => "RT_IDS"
}

Any assistance will be truly appreciated with this challenge.

2

I just tried that on http://grokdebug.herokuapp.com/ and it doesn't work.
I'd start by hitting up that site and then back tracking till you find the problem.

3

Thank you for the response, it is quite interesting as I was using http://grokconstructor.appspot.com/do/construction and all was matching here, however in http://grokdebug.herokuapp.com/ this was not the case. I have updated the filter and now it is matching in both tools, however the challenge is still current, here is the latest filer:
%{CISCOTIMESTAMP} %{IP} %{WORD:Nr} %{SYSLOG5424PRI}1 %{TIMESTAMP_ISO8601} %{IP} %{WORD} - - - %{TIMESTAMP_ISO8601:Date} %{CISCOTAG} RT_IDS - RT_SCREEN_TCP_LS %{SYSLOG5424SD}

4

It looks like the date is the challenge in the filter, could someone assist in validating if the following is correct:

Date Example:
2016-07-28T03:51:34.068+01:00

Filter Applied:
"yyyy-MM-dd'T'HH:mm:ss.SSSz"

Thank you

5

Just use the TIMESTAMP_ISO8601 pattern :slight_smile:

6

Thank you for that, however let me provide you with the configuration sections I am struggling with:
if "RT_IDS" in [tags] {
grok {
match => ["messages", "%{CISCOTIMESTAMP} %{IP} %{WORD:Nr} %{SYSLOG5424PRI}1 %{TIMESTAMP_ISO8601} %{IP} %{WORD} - - - %{TIMESTAMP_ISO8601:Date} %{CISCOTAG} RT_IDS - RT_SCREEN_TCP_LS %{SYSLOG5424SD}"]
add_tag => "RT_IDS"
}
date {
locale => "en"
match => ["Date",
"yyyy-mm-dd'T'HH:mm:ss.SSSZ",
"ISO8601"]
timezone => "Africa/Windhoek"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}

7

Log:
Jul 22 03:30:10 192.29.58.44 400 <11>1 2016-07-22T03:17:29+01:00 10.31.12.12 1 - - - 2016-07-22T03:29:14.649+01:00 ABC-0000-WWWW03 RT_IDS - RT_SCREEN_TCP_LS [junos@1111.1.1.1.1.88 logical-system-name="TEST-INTERNET" attack-name="TCP sweep!" source-address="58.18.186.131" source-port="45633" destination-address="106.225.87.224" destination-port="22" source-zone-name="INTERNET-TEST" interface-name="xe-1/0/13.9" action="drop"]