Logstash "_grokparsefailure" in output

Elastic Stack Logstash
1

Hi, I am trying to execute a simple logstash config by using the grok patterns. Following is the total.conf file:

input {
tcp {
    port => 5000
    type => meter
}
udp {
    port => 5000
    type => meter
}

}

filter {
if [type] == "meter" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => '%{TIMESTAMP_ISO8601:somestamp} %{HOSTNAME:someId} %{GREEDYDATA:someevents} %{GREEDYDATA:somemode}' }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => ["timestamp", "ISO8601"]
timezone => "US/Central"
target => "@timestamp"
add_tag => [ "timestampMatched" ]
}
}
}

output {
stdout { codec => rubydebug }
}

I start the logstash using:

/usr/share/logstash/bin/logstash --debug -f /tmp/total.conf
Following is the sample input:
2017-02-09T08:23:46.577-0600 xxxxxxxxxxxxxxxxxxx 563
2017-02-09T08:23:58.610-0600 xxxxxxxxxxxxxxxxxxx 1127
2017-02-09T08:24:10.642-0600 xxxxxxxxxxxxxxxxxxx 1287
2017-02-09T08:24:22.679-0600 xxxxxxxxxxxxxxxxxxx 536
2017-02-09T08:24:34.711-0600 xxxxxxxxxxxxxxxxxxx 1808

And i get the output as:

{
"@timestamp" => 2017-02-09T14:18:05.795Z,
"port" => 56204,
"@version" => "1",
"host" => "0:0:0:0:0:0:0:1",
"message" => "2017-02-09T08:17:07.852-0600 xxxxxxxxxxxxxx 1296\r",
"type" => "meter",
"tags" => [
[0] "_grokparsefailure"
]
}

The pattern works good when applied in the grok debugger and gives me the output as:

{
"timestamp": [
[
"2017-02-07T12:59:12.568-06:00"
]
],
"hostname": [
[
"xxxxxxxxxxxxxxx"
]
],
"events": [
[
"1633"
]
]
}

But its not working with logstash.
What am i doing wrong?

-Thanks

2

Your grok expression expects four fields but your input only provides three.

Also, while unrelated to your problem you're extracting the timestamp to the somestamp field but that's not the field you're feeding the date filter with.

3

@magnusbaeck, My Bad, that worked!
I didnt see that before. The problem is fixed when i changed the number of parameters.
But when i run this configuration within the conf.d with other files the logstash-plain.log is flooded with

Ruby exception occurred: undefined method `%' for nil:NilClass

Is there a way i can debug why this error is occurring?

4

You have a ruby filter in your configuration?

6

Here's the problematic piece of code:

event.get('offset') % 1000

If the event doesn't have an offset field, event.get will return nil and Ruby will complain that you can't apply the % operator on a nil value.

7

I placed my filter.conf file after the file containing ruby filter and it worked.
Thank you @magnusbaeck.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.