Grok parse failures .. Grok syntax

ranganath_nangineni · October 11, 2018, 3:28pm

Hi,

I have matching records for the below grok pattern
[%{TIMESTAMP_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]

But the logstash is erroring on the below syntax: If I comment this line, the pipelines are working fine.

match => { "message" => "%[%{TIMESTAMP_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]" }

Error:

[2018-10-11T15:26:29,677][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 25, column 202 (byte 1076) after filter {\n\n############### Start of OHS Handlers ################\n if [fields][log_type] == "ohsa" or [fields][log_type] == "ohs" {\n if [fields][app] == "comm" {\ngrok {\n break_on_match => "true"\n match => { "message" => "%{IPORHOST:clientip} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] %{NOTSPACE:ecid} \"%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:trueclientip} %{NOTSPACE:asntmp}" }\n match => {"message" => "%{COMBINEDAPACHELOG}"}\n }\n}\n############################## RUNDECK LOGS ######################### \n else if [fields][app] == "rundeck" {\ngrok {\n patterns_dir => ["/usr/share/logstash/patterns"]\n break_on_match => "true"\n match => { "message" => "%{RUNDECKLOG}"}\n match => { "message" => "%\[%{TIMESTAMP_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{GREEDYDATA:STATE}\] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:inblock in converge_state'"]}

Any help is much apprecitated?

ranganath_nangineni · October 11, 2018, 4:08pm

Sample log record:

[2018-10-10 22:20:22,372] xyz@xyz.com finish [2049:succeeded] Monitoring xyz@xyz.com/ - "-/BigIP Healthcheck - ADC"[2d7c9767-16cc-4566-ac77-ab972048d5ff]

Makra · October 12, 2018, 3:54am

The grok that you have mentioned did not work in https://grokdebug.herokuapp.com/ , Also you need to escaped characters like [ or - for logstash to parse the field properly.

I have tested the log with the following GROK and it did worked.

\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{NOTSPACE:STATE}\] %{NOTSPACE:PROJECT} %{DATA:USER}\/ \- \"-%{NOTSPACE:PATH} %{GREEDYDATA:JOBNAME}\"\[%{NOTSPACE:USERID}\]

ranganath_nangineni · October 12, 2018, 4:47am

This is the pattern that I tested and it works on http://grokconstructor.appspot.com
But the same is not working in logstash. How do I place the message => part for this ?

[%{TIMESTAMP_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]

Makra · October 12, 2018, 4:56am

The error seems to be in line number 25 and Logstash combines all the files in your config directory into a single file. When there's an error, you're getting line and position information from the merged config file.

Run the following command to combine all conf file into a single file and then check line number 25.

cat /etc/logstash/conf.d/* > /tmp/single.conf

ranganath_nangineni · October 12, 2018, 5:06am

This is the 25th line.

match => { "message" => "%[%{TIMESTAMP_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]" }

ranganath_nangineni · October 12, 2018, 5:18am

The highlighted portion is the problematic area.
I need a help in creating this match => pattern

Makra · October 12, 2018, 5:24am

I tested the grok against the log, it produces compiler error.

https://imgur.com/a/gDX3Q4M

ranganath_nangineni · October 12, 2018, 5:27am

Some how the grok that you entered is not correct. Please see the Grok that I am using from the picture uploaded.

Makra · October 12, 2018, 5:31am

Instead of double quotes, try using single quotes around grok also escape the special characters with a \

{ "message" => '[%{TIMESTAMP_ISO8601:timestamp}] .......{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]' }

ranganath_nangineni · October 12, 2018, 5:45am

I used it like this, The error is not there now but the getting the beats_input_codec_plain_applied, _grokparsefailure .

The records are not processed as expected. Anywhere I am missing the special characters?

Makra · October 12, 2018, 5:47am

The _grokparsefailure means the the grok pattern is not matched against the logs. Can you post one logline and the filter again ?

ranganath_nangineni · October 12, 2018, 5:51am

[2018-10-11 06:01:00,059] xyz@xyz.com start [2972:running] Monitoring xyz@xyz.com/- "-/BigIP Healthcheck - ADC"[2d7c9767-16cc-4566-ac77-ab972048d5ff]

Filter used:

Makra · October 12, 2018, 5:52am

Please in text

ranganath_nangineni · October 12, 2018, 5:56am

match => { 'message' => '\[%{TIMESTAMP_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{GREEDYDATA:STATE}\] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"\[%{GREEDYDATA:uuid}\]' }

Makra · October 12, 2018, 5:58am

There is an error in the grok, please check it again https://grokdebug.herokuapp.com/

ranganath_nangineni · October 12, 2018, 6:08am

The error could be that the grokdebugger that "https://grokdebug.herokuapp.com/" is using not having GROK pattern for HTTPDUSER.

Please test the same on "http://grokconstructor.appspot.com/do/match#result"

It is working fine without any issues.

Makra · October 12, 2018, 6:40am

I didn't get any _grokparse failure with the following pattern.

match => { "message" => '\[%{TIMESTAMP_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"\[%{GREEDYDATA:uuid}\]' }

12:06:39.702 [[main]-pipeline-manager] DEBUG logstash.filters.grok - **Grok compiled OK** {:pattern=>"\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} \"%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}\"\\[%{GREEDYDATA:uuid}\\]", :expanded_pattern=>"\\[(?<TIMESTAMP_ISO8601:timestamp>(?:(
?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T ](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\] (?<HTTPDUSER:EVENTUSER>(?:(?:[a-zA-Z][a-zA-Z0-9_.+-=:]+)@(?:\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\
\b)))|(?:(?:[a-zA-Z0-9._-]+))) (?<WORD:EVENT>\\b\\w+\\b) [(?<NUMBER:ID>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))))):(?<GREEDYDATA:STATE>.*)] (?<GREEDYDATA:project>.*) (?<HTTPDUSER:USER>(?:(?:[a-zA-Z][a-zA-Z0-9_.+-=:]+)@(?:\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\\b)))|(?:(?:[a-zA-Z0-9._-]+)))/ (?<HTTPDUSER:abortedby>(?:(?:[a-z

ranganath_nangineni · October 12, 2018, 6:43am

This _grokparse failure is added as a "tag" to the logstash output record.
The message is not prased completely.

ranganath_nangineni · October 12, 2018, 8:12am

I am able to resolve the issue by adjusting the spaces in the grok pattern. The same pattern with the space adjustments worked fine .

I am good now.

Thanks Makra.