Grok pattern working fine in grok debugger but the same pattern is not working when running with logstash

Elastic Stack Logstash
1

Hi, I am using below pattern to parse below Log, the pattern is working fine grok debugger but failing when running in logstash. Please help

Grok Pattern: %{DATA:hour}:%{DATA:minute}:%{DATA:second},%{DATA:milisecond} %{LOGLEVEL:loglevel} %{WORD:service}:%{INT:id} \- %{GREEDYDATA:msgbody}

Log: 16:05:05,972 INFO ZooKeeper:438 - Initiating client connection, connectString:2181=1-2-3-4.xxxxxx.com sessionTimeout=60000 watcher=hconnection-0x635f, quorum:2181=1-2-3-4.xxxx.com, baseZNode=/hbase

2

Never use more than one DATA or GREEDYDATA in the same expression. Build the expression gradually and be as exact as you can. Start with the simplest possible expression, ^%{TIME:time} and build from there.

3

Hi Magnus, I have tried by adding one by one pattern but all the time it is giving "_grokparsefailure".

Pattern: %{TIME:time} %{LOGLEVEL:loglevel} %{WORD:class}:%{INT:id} - %{GREEDYDATA:msgbody}

My logstash.conf file contents below:

input {
beats {
port => 5000
}
}
filter {
if [ "source" ] == [ "/govind/rest.txt" ] {
grok {
match => {"message" => "%{TIME:time} %{LOGLEVEL:loglevel} %{DATA:class} - %{COMMONAPACHELOG:msgbody}"}
remove_field => [ "msgbody" ]
remove_field => [ "input_type" ]
remove_field => [ "auth" ]
remove_field => [ "ident" ]
}
}
else {
grok {
match => {"message" => "%{TIME:time} %{LOGLEVEL:loglevel} %{WORD:class}:%{INT:id} - %{GREEDYDATA:msgbody}"}
}
}
}
output {
stdout { codec => rubydebug }
}

First grok filter in if condition is working fine but else grok filter is failing all the time

4

Hi Magnus, I have tried by adding one by one pattern but all the time it is giving "_grokparsefailure".

You mean it's failing even with just ^%{TIME:time} as the pattern? Please show an example of such a message (use your stdout output).

5

Yes Magnus it is failing for time pattern aswell

stdoutput:

{
"@timestamp" => 2017-12-12T13:55:17.505Z,
"offset" => 5209566,
"@version" => "1",
"beat" => {
"hostname" => "1-2-3-4.xxxxx.com",
"name" => "1-2-3-4.xxxx.com",
"version" => "5.5.0"
},
"input_type" => "log",
"host" => "1-2-3-4.xxxx.com",
"source" => "/govind/app.txt",
"message" => "13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server 1-2-3-4.xxxx.com/1.2.3.4:2181, sessionid = 0x160322a70x4e, negotiated timeout = 60000",
"type" => "log",
"tags" => [
[0] "beats_input_codec_plain_applied",
[1] "_grokparsefailure"
]
}

6

Works fine here:

$ cat test.config 
input { stdin { } }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => {
      "message" => "^%{TIME:time}"
    }
  }
}
$ echo '13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server 1-2-3-4.xxxx.com/1.2.3.4:2181, sessionid = 0x160322a70x4e, negotiated timeout = 60000' | /opt/logstash/bin/logstash -f test.config 
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "13:55:15,295 INFO ClientCnxn:1235 - Session establishment complete on server 1-2-3-4.xxxx.com/1.2.3.4:2181, sessionid = 0x160322a70x4e, negotiated timeout = 60000",
      "@version" => "1",
    "@timestamp" => "2017-12-12T15:06:52.360Z",
          "host" => "lnxolofon",
          "time" => "13:55:15,295"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

Perhaps you have another grok filter in another file in /etc/logstash/conf.d or wherever your configuration files reside?

7

It is working fine now. But when i tired to use the same pattern in my below else condition it is failing. First grok filter in if else condition is working but the else condition grok filter is not working. Is there any issue with my if else syntax?

input {
beats {
port => 5000
}
}
filter {
if [ "source" ] == [ "/govind/app.txt" ] {
grok {
match => {"message" => "%{TIME:time} %{LOGLEVEL:loglevel} %{DATA:class} - %{COMMONAPACHELOG:msgbody}"}
remove_field => [ "msgbody" ]
remove_field => [ "input_type" ]
remove_field => [ "auth" ]
remove_field => [ "ident" ]
}
}
else {
grok {
match => {"message" => "%{TIME:time} %{LOGLEVEL:loglevel} %{WORD:service}:%{INT:serviceid} - %{GREEDYDATA:msgbody}"}
}
}
}
output {
stdout { codec => rubydebug }
}

8 
if [ "source" ] == [ "/logsForShip/fraudqm_kafka_rest/rest.txt" ] {

This doesn't do what you think it does. Change to this:

if [source] == "/logsForShip/fraudqm_kafka_rest/rest.txt" {
9

Changed but the second else condition grok filter is not working. when i run the same grok filter in separate config file with out if condition it is working

10

And what does the stdout output produce for such an event?

11

Hi Magnus,

For above query, i am able to fix it with your guidance. But i am facing issue when doing below comparison with float value in grok filter

Code snippet:

output {

if [totaldelay] >= 0.500 {
email {
to => 'govinda.rao@xxxx.com'
from => 'job@localhost.com'
subject => 'Alert - batch is running longer. Please check...'
body => "%{message}"
domain => 'mail.localhost.com'
port => 25
}
}
}

Error:

ArgumentError: comparison of Float with nil failed
>= at org/jruby/RubyFloat.java:595
output_func at (eval):128
output_batch at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:293
each at org/jruby/RubyArray.java:1613
inject at org/jruby/RubyEnumerable.java:852
output_batch at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:287
worker_loop at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232
start_workers at /home/govind/logstash-2.4.1/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201

12

The error indicates that the totaldelay field isn't set. Replace the conditional with:

if [totaldelay] and [totaldelay] >= 0.500 {
13

Thanks Magnus it worked. Is there a way i can implement in the above output code snippet that, if [totaldelay] >=0.500 continuously for 15 minutes then send an email alert. Please help i am not getting any clue on this

14

Logstash isn't an alerting tool and you're better off using something else for that.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.