# Grok pattern working fine in grok debugger but the same pattern is not working when running with logstash **URL:** \ **Category:** Logstash\ **Created:** [December 11, 2017, 5:10pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150 "2017-12-11T17:10:54Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [December 11, 2017, 5:10pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/1 "2017-12-11T17:10:54Z")
Hi, I am using below pattern to parse below Log, the pattern is working fine grok debugger but failing when running in logstash. Please help **Grok Pattern:** %{DATA:hour}:%{DATA:minute}:%{DATA:second},%{DATA:milisecond} %{LOGLEVEL:loglevel} %{WORD:service}:%{INT:id} \- %{GREEDYDATA:msgbody} **Log** : 16:05:05,972 INFO ZooKeeper:438 - Initiating client connection, [connectString:2181=1-2-3-4.xxxxxx.com](http://connectString:2181=1-2-3-4.xxxxxx.com) sessionTimeout=60000 watcher=hconnection-0x635f, [quorum:2181=1-2-3-4.xxxx.com](http://quorum:2181=1-2-3-4.xxxx.com), baseZNode=/hbase ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 12, 2017, 6:37am UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/2 "2017-12-12T06:37:16Z")
Never use more than one DATA or GREEDYDATA in the same expression. Build the expression gradually and be as exact as you can. Start with the simplest possible expression, `^%{TIME:time}` and build from there. ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [December 12, 2017, 2:00pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/3 "2017-12-12T14:00:50Z")
Hi Magnus, I have tried by adding one by one pattern but all the time it is giving "\_grokparsefailure". Pattern: %{TIME:time} %{LOGLEVEL:loglevel} %{WORD:class}:%{INT:id} - %{GREEDYDATA:msgbody} My logstash.conf file contents below: input { beats { port =\> 5000 } } filter { if ["source"] == ["/govind/rest.txt"] { grok { match =\> {"message" =\> "%{TIME:time} %{LOGLEVEL:loglevel} %{DATA:class} - %{COMMONAPACHELOG:msgbody}"} remove\_field =\> ["msgbody"] remove\_field =\> ["input\_type"] remove\_field =\> ["auth"] remove\_field =\> ["ident"] } } else { grok { match =\> {"message" =\> "%{TIME:time} %{LOGLEVEL:loglevel} %{WORD:class}:%{INT:id} - %{GREEDYDATA:msgbody}"} } } } output { stdout { codec =\> rubydebug } } First grok filter in if condition is working fine but else grok filter is failing all the time ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 12, 2017, 2:29pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/4 "2017-12-12T14:29:02Z")
> Hi Magnus, I have tried by adding one by one pattern but all the time it is giving "\_grokparsefailure". You mean it's failing even with just `^%{TIME:time}` as the pattern? Please show an example of such a message (use your stdout output). ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [December 12, 2017, 2:50pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/5 "2017-12-12T14:50:07Z")
Yes Magnus it is failing for time pattern aswell stdoutput: { "@timestamp" =\> 2017-12-12T13:55:17.505Z, "offset" =\> 5209566, "@version" =\> "1", "beat" =\> { "hostname" =\> "[1-2-3-4.xxxxx.com](http://1-2-3-4.xxxxx.com)", "name" =\> "[1-2-3-4.xxxx.com](http://1-2-3-4.xxxx.com)", "version" =\> "5.5.0" }, "input\_type" =\> "log", "host" =\> "[1-2-3-4.xxxx.com](http://1-2-3-4.xxxx.com)", "source" =\> "/govind/app.txt", "message" =\> "13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server [1-2-3-4.xxxx.com/1.2.3.4:2181](http://1-2-3-4.xxxx.com/1.2.3.4:2181), sessionid = 0x160322a70x4e, negotiated timeout = 60000", "type" =\> "log", "tags" =\> [ [0] "beats\_input\_codec\_plain\_applied", [1] "\_grokparsefailure" ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 12, 2017, 3:08pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/6 "2017-12-12T15:08:03Z")
Works fine here: ```nohighlight $ cat test.config input { stdin { } } output { stdout { codec => rubydebug } } filter { grok { match => { "message" => "^%{TIME:time}" } } } $ echo '13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server 1-2-3-4.xxxx.com/1.2.3.4:2181, sessionid = 0x160322a70x4e, negotiated timeout = 60000' | /opt/logstash/bin/logstash -f test.config Settings: Default pipeline workers: 8 Pipeline main started { "message" => "13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server 1-2-3-4.xxxx.com/1.2.3.4:2181, sessionid = 0x160322a70x4e, negotiated timeout = 60000", "@version" => "1", "@timestamp" => "2017-12-12T15:06:52.360Z", "host" => "lnxolofon", "time" => "13:55:15,295" } Pipeline main has been shutdown stopping pipeline {:id=>"main"} ``` Perhaps you have another grok filter in another file in /etc/logstash/conf.d or wherever your configuration files reside? ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [December 12, 2017, 5:12pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/7 "2017-12-12T17:12:48Z")
It is working fine now. But when i tired to use the same pattern in my below else condition it is failing. First grok filter in if else condition is working but the else condition grok filter is not working. Is there any issue with my if else syntax? input { beats { port =\> 5000 } } filter { if ["source"] == ["/govind/app.txt"] { grok { match =\> {"message" =\> "%{TIME:time} %{LOGLEVEL:loglevel} %{DATA:class} - %{COMMONAPACHELOG:msgbody}"} remove\_field =\> ["msgbody"] remove\_field =\> ["input\_type"] remove\_field =\> ["auth"] remove\_field =\> ["ident"] } } else { grok { match =\> {"message" =\> "%{TIME:time} %{LOGLEVEL:loglevel} %{WORD:service}:%{INT:serviceid} - %{GREEDYDATA:msgbody}"} } } } output { stdout { codec =\> rubydebug } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 12, 2017, 7:29pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/8 "2017-12-12T19:29:40Z")
> ``` > if ["source"] == ["/logsForShip/fraudqm_kafka_rest/rest.txt"] { > > ``` This doesn't do what you think it does. Change to this: ``` if [source] == "/logsForShip/fraudqm_kafka_rest/rest.txt" { ``` ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [December 13, 2017, 11:20am UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/9 "2017-12-13T11:20:57Z")
Changed but the second else condition grok filter is not working. when i run the same grok filter in separate config file with out if condition it is working ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 13, 2017, 12:06pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/10 "2017-12-13T12:06:51Z")
And what does the stdout output produce for such an event? ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [January 3, 2018, 7:49pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/11 "2018-01-03T19:49:30Z")
Hi Magnus, For above query, i am able to fix it with your guidance. But i am facing issue when doing below comparison with float value in grok filter **Code snippet:** output { if [totaldelay] \>= 0.500 { email { to =\> 'govinda.rao@xxxx.com' from =\> 'job@localhost.com' subject =\> 'Alert - batch is running longer. Please check...' body =\> "%{message}" domain =\> '[mail.localhost.com](http://mail.localhost.com)' port =\> 25 } } } **Error:** ArgumentError: comparison of Float with nil failed \>= at org/jruby/RubyFloat.java:595 output\_func at (eval):128 output\_batch at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:293 each at org/jruby/RubyArray.java:1613 inject at org/jruby/RubyEnumerable.java:852 output\_batch at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:287 worker\_loop at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232 start\_workers at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 3, 2018, 9:12pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/12 "2018-01-03T21:12:35Z")
The error indicates that the `totaldelay` field isn't set. Replace the conditional with: ``` if [totaldelay] and [totaldelay] >= 0.500 { ``` ---
**Author:** ![govinda.rao](https://avatars.discourse-cdn.com/v4/letter/g/f14d63/32.png) [@govinda.rao](https://discuss.elastic.co/u/govinda.rao)\ **Post date:** [January 4, 2018, 1:07pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/13 "2018-01-04T13:07:50Z")
Thanks Magnus it worked. Is there a way i can implement in the above output code snippet that, if [totaldelay] \>=0.500 continuously for 15 minutes then send an email alert. Please help i am not getting any clue on this ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 4, 2018, 2:30pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/14 "2018-01-04T14:30:43Z")
Logstash isn't an alerting tool and you're better off using something else for that. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2018, 2:30pm UTC](https://discuss.elastic.co/t/grok-pattern-working-fine-in-grok-debugger-but-the-same-pattern-is-not-working-when-running-with-logstash/111150/15 "2018-02-01T14:30:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.