# Grok parse failures .. Grok syntax **URL:** \ **Category:** Logstash\ **Created:** [October 11, 2018, 3:28pm UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083 "2018-10-11T15:28:33Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 11, 2018, 3:28pm UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/1 "2018-10-11T15:28:33Z")
Hi, I have matching records for the below grok pattern [%{TIMESTAMP\_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}] But the logstash is erroring on the below syntax: If I comment this line, the pipelines are working fine. match =\> { "message" =\> "%[%{TIMESTAMP\_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]" } # Error: [2018-10-11T15:26:29,677][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, {, } at line 25, column 202 (byte 1076) after filter {\n\n############### Start of OHS Handlers ################\n if [fields][log\_type] == "ohsa" or [fields][log\_type] == "ohs" {\n if [fields][app] == "comm" {\ngrok {\n break\_on\_match =\> "true"\n match =\> { "message" =\> "%{IPORHOST:clientip} %{USER:ident} %{NOTSPACE:auth} \[%{HTTPDATE:timestamp}\] %{NOTSPACE:ecid} \"%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:trueclientip} %{NOTSPACE:asntmp}" }\n match =\> {"message" =\> "%{COMBINEDAPACHELOG}"}\n }\n}\n############################## RUNDECK LOGS ######################### \n else if [fields][app] == "rundeck" {\ngrok {\n patterns\_dir =\> ["/usr/share/logstash/patterns"]\n break\_on\_match =\> "true"\n match =\> { "message" =\> "%{RUNDECKLOG}"}\n match =\> { "message" =\> "%\[%{TIMESTAMP\_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{GREEDYDATA:STATE}\] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:149:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:22:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:38:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309:in`block in converge\_state'"]} Any help is much apprecitated? ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 11, 2018, 4:08pm UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/2 "2018-10-11T16:08:55Z")
Sample log record: [2018-10-10 22:20:22,372] [xyz@xyz.com](mailto:xyz@xyz.com) finish [2049:succeeded] Monitoring [xyz@xyz.com](mailto:xyz@xyz.com)/ - "-/BigIP Healthcheck - ADC"[2d7c9767-16cc-4566-ac77-ab972048d5ff] ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 3:54am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/3 "2018-10-12T03:54:07Z")
The grok that you have mentioned did not work in [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) , Also you need to escaped characters like [ or - for logstash to parse the field properly. I have tested the log with the following GROK and it did worked. `\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{NOTSPACE:STATE}\] %{NOTSPACE:PROJECT} %{DATA:USER}\/ \- \"-%{NOTSPACE:PATH} %{GREEDYDATA:JOBNAME}\"\[%{NOTSPACE:USERID}\]` ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 4:47am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/4 "2018-10-12T04:47:25Z")
> [@Makra](#): > > [%{TIMESTAMP\_ISO8601:timestamp}] %{DATA:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{NOTSPACE:STATE}] %{NOTSPACE:PROJECT} %{DATA:USER}/ - "-%{NOTSPACE:PATH} %{GREEDYDATA:JOBNAME}"[%{NOTSPACE:USERID}] This is the pattern that I tested and it works on [http://grokconstructor.appspot.com](http://grokconstructor.appspot.com) But the same is not working in logstash. How do I place the message =\> part for this ? [%{TIMESTAMP\_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}] ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 4:56am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/5 "2018-10-12T04:56:06Z")
> [@ranganath\_nangineni](#): > > :message=\>"Expected one of #, {, } at line 25, column 202 (byte 1076) after filter The error seems to be in line number 25 and Logstash combines all the files in your config directory into a single file. When there's an error, you're getting line and position information from the merged config file. Run the following command to combine all conf file into a single file and then check line number 25. ```auto cat /etc/logstash/conf.d/* > /tmp/single.conf ``` ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:06am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/6 "2018-10-12T05:06:29Z")
> [@Makra](#): > > cat /etc/logstash/conf.d/\* \> /tmp/single.conf This is the 25th line. match =\> { "message" =\> "%[%{TIMESTAMP\_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}]" } ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:18am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/7 "2018-10-12T05:18:04Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1e882d15fae981d691cc53858fd64ba573362c7c.png) The highlighted portion is the problematic area. I need a help in creating this match =\> pattern ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 5:24am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/8 "2018-10-12T05:24:23Z")
> [@ranganath\_nangineni](#): > > [2018-10-10 22:20:22,372] [xyz@xyz.com](mailto:xyz@xyz.com) finish [2049:succeeded] Monitoring [xyz@xyz.com](mailto:xyz@xyz.com)/ - "-/BigIP Healthcheck - ADC"[2d7c9767-16cc-4566-ac77-ab972048d5ff] I tested the grok against the log, it produces compiler error. [https://imgur.com/a/gDX3Q4M](https://imgur.com/a/gDX3Q4M) ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:27am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/9 "2018-10-12T05:27:32Z")
Some how the grok that you entered is not correct. Please see the Grok that I am using from the picture uploaded. ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 5:31am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/10 "2018-10-12T05:31:17Z")
Instead of double quotes, try using single quotes around grok also escape the special characters with a \ { "message" =\> **'** [%{TIMESTAMP\_ISO8601:timestamp}] .......{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}] **'** } ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:45am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/11 "2018-10-12T05:45:13Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9d109f2321f8fc0d1e08b4f3f5af2bbae3916b66.png) I used it like this, The error is not there now but the getting the beats\_input\_codec\_plain\_applied, \_grokparsefailure . The records are not processed as expected. Anywhere I am missing the special characters? ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 5:47am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/12 "2018-10-12T05:47:01Z")
The \_grokparsefailure means the the grok pattern is not matched against the logs. Can you post one logline and the filter again ? ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:51am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/13 "2018-10-12T05:51:47Z")
[2018-10-11 06:01:00,059] [xyz@xyz.com](mailto:xyz@xyz.com) start [2972:running] Monitoring [xyz@xyz.com](mailto:xyz@xyz.com)/- "-/BigIP Healthcheck - ADC"[2d7c9767-16cc-4566-ac77-ab972048d5ff] Filter used: ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9d109f2321f8fc0d1e08b4f3f5af2bbae3916b66.png) ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 5:52am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/14 "2018-10-12T05:52:20Z")
Please in text ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 5:56am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/15 "2018-10-12T05:56:01Z")
`match => { 'message' => '\[%{TIMESTAMP_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} \[%{NUMBER:ID}:%{GREEDYDATA:STATE}\] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"\[%{GREEDYDATA:uuid}\]' }` ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 5:58am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/16 "2018-10-12T05:58:11Z")
> [@ranganath\_nangineni](#): > > [%{TIMESTAMP\_ISO8601:timestamp}] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"[%{GREEDYDATA:uuid}] There is an error in the grok, please check it again [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 6:08am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/17 "2018-10-12T06:08:53Z")
The error could be that the grokdebugger that "[https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)" is using not having GROK pattern for HTTPDUSER. Please test the same on "[http://grokconstructor.appspot.com/do/match#result](http://grokconstructor.appspot.com/do/match#result)" It is working fine without any issues. ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/b/7b3962c747b3912ff74b66ff29b9f6e135d19fcf.png) ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [October 12, 2018, 6:40am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/18 "2018-10-12T06:40:09Z")
I didn't get any \_grokparse failure with the following pattern. `match => { "message" => '\[%{TIMESTAMP_ISO8601:timestamp}\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} "%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}"\[%{GREEDYDATA:uuid}\]' }` ``` 12:06:39.702 [[main]-pipeline-manager] DEBUG logstash.filters.grok - **Grok compiled OK** {:pattern=>"\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{HTTPDUSER:EVENTUSER} %{WORD:EVENT} [%{NUMBER:ID}:%{GREEDYDATA:STATE}] %{GREEDYDATA:project} %{HTTPDUSER:USER}/ %{HTTPDUSER:abortedby} \"%{GREEDYDATA:PATH}/%{GREEDYDATA:jobName}\"\\[%{GREEDYDATA:uuid}\\]", :expanded_pattern=>"\\[(?(?:( ?>\\d\\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[T](?:(?:2[0123]|[01]?[0-9])):?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|[+-](?:(?:2[0123]|[01]?[0-9]))(?::?(?:(?:[0-5][0-9])))))?)\\] (?(?:(?:[a-zA-Z][a-zA-Z0-9_.+-=:]+)@(?:\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\ \b)))|(?:(?:[a-zA-Z0-9._-]+))) (?\\b\\w+\\b) [(?(?:(?:(?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))))):(?.*)] (?.*) (?(?:(?:[a-zA-Z][a-zA-Z0-9_.+-=:]+)@(?:\\b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\\.?|\\b)))|(?:(?:[a-zA-Z0-9._-]+)))/ (?(?:(?:[a-z ``` ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 6:43am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/19 "2018-10-12T06:43:01Z")
This \_grokparse failure is added as a "tag" to the logstash output record. The message is not prased completely. ---
**Author:** ![ranganath\_nangineni](https://avatars.discourse-cdn.com/v4/letter/r/3ec8ea/32.png) [@ranganath\_nangineni](https://discuss.elastic.co/u/ranganath_nangineni)\ **Post date:** [October 12, 2018, 8:12am UTC](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083/20 "2018-10-12T08:12:19Z")
I am able to resolve the issue by adjusting the spaces in the grok pattern. The same pattern with the space adjustments worked fine . I am good now. Thanks Makra. [Next page](https://discuss.elastic.co/t/grok-parse-failures-grok-syntax/152083.md?page=2)