Grokparse failure - but grokdebugger works

Elastic Stack Logstash
1

hi all

I have the following log line:

{"rule":{"level":12,"comment":"apache error tagged by modsecurity","sidid":130401},"location":"(cli-git) 52.16.98.219->/var/log/apache2/error.log","full_log":"[Tue May 05 09:09:08.901326 2015] [:error] [pid 7085] [client 212.48.71.196] ModSecurity: Warning. Pattern match "(?i:\\\\bor\\\\b ?(?:\\\\d{1,10}|[\\\\'\\"][^=]{1,10}[\\\\'\\"]) ?[=<>]+|(?i:'\\\\s+x?or\\\\s+.{1,20}[+\\\\-!<>=])|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')|\\\\b(?i:x?or)\\\\b\\\\s+(\\\\d{1,10}|'[^=]{1,10}')\\\\s*?[=<>])" at ARGS:id. [file "/usr/share/modsecurity-crs/activated_rules/modsecurity_crs_41_sql_injection_attacks.conf"] [line "133"] [id "959071"] [rev "2"] [msg "SQL Injection Attack"] [data "Matched Data: ' or true -- found within ARGS:id: ' or true -- "] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "8"] [tag "OWASP_CRS/WEB_ATTACK/SQL_INJECTION"] [tag "WASCTC/WASC-19"] [tag "OWASP_TOP_10/A1"] [tag "OWASP_AppSensor/CIE1"] [tag "PCI/6.5.2"] [hostname "52.16.98.219"] [uri "/dvwa/vulnerabilities/sqli/"] [unique_id "VUiItH8AAAEAAButAp4AAAAE"]"}

I am using the following filter and pattern. My input is stdin and codec => json.

filter {
grok {
patterns_dir => "./patterns"
match => ["full_log", "%{FULLDATE:thedate}.+%{CLIENT:clientip}.+%{ATTACKVECTOR:attack}.+%{HOSTNAME:thehost}%{GREEDYDATA:therest}"]
}

In patterns/test:

FULLDATE %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}
ATTACKVECTOR msg \".+?\"
CLIENT client %{IPV4}
HOSTNAME hostname \"%{IPV4}\"

But the output (using stdout and codec => rubydebug) results in "full_log" not being parsed and the line:

[0] "_grokparsefailure"

I've tested the filter & pattern on grokdebug and it works. I don't know why it doesn't work in real-life. I'm running OpenJDK java version "1.7.0_79" on Ubuntu 14.04 with Logstash 1.4.2.

Thanks

Jay

2

perhaps any whitespaces in the actual grok definition file?

3

also shouldn't you be matching on "message" field instead?
and how are you matchin

{"rule":{"level":12,"comment":"apache error tagged by modsecurity","sidid":130401},"location":"(cli-git) 52.16.98.219->/var/log/apache2/error.log","full_log":"[

?

4

also shouldn't you be matching on "message" field instead?
and how are you matchin

This should be fine because of codec => json.