Logstash filter event.get() is hard to debug

[John_ax]

Hi, I am using logstash 5.1.1, got lots of ERROR logs, so I try this:

echo '{"msg":"xxxx", "@timestamp":"2017-08-28T08:16:54.943Z", "service":"x.y.z", "host":"ubuntu","timestamp":1503908214}' |
./logstash   -e  'input { stdin { codec => json} } output { stdout { codec => rubydebug } } filter {ruby {
code => "event.set('timestamp', event.get('timestamp') * 1000) if event.get('timestamp') < 10000000000"
	}
	date {
	    match => ["timestamp", "UNIX_MS"]
	    add_field => { "[@metadata][index]" => "%{service}_%{[@metadata][index_day]}"}
	    remove_field => ["@timestamp","@version","source","file","offset","beat","topic","input_type","id"]
	}
}
'

but keep failing:

[2017-08-29T18:02:11,268][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined local variable or method `timestamp' for #<LogStash::Filters::Ruby:0x506d34a3>
{
          "msg" => "xxxx",
      "service" => "x.y.z",
         "host" => "ubuntu",
    "timestamp" => 1503908214,
         "tags" => [
        [0] "_rubyexception"
    ]
}
[2017-08-29T18:02:14,196][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

the filter is so hard to debug, in Fact the filter config works fine in the realworld

[John_ax]

sometime I got this:

[2017-08-29T18:17:23,336][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:23,587][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:24,155][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:24,233][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:24,326][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:24,925][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:25,078][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass
[2017-08-29T18:17:25,706][ERROR][logstash.filters.ruby    ] Ruby exception occurred: undefined method `<' for nil:NilClass

still have no idea what happen.....

[magnusbaeck (Magnus Bäck)]

The first problem is caused by you having a single-quoted string inside your single quoted string:

logstash -e '... code => "event.set('timestamp', ...

Escaping the inner single quotes should work.

Ruby exception occurred: undefined method `<' for nil:NilClass

Given

if event.get('timestamp') < 10000000000

in your code I'd say the problem is that the event doesn't have a timestamp field (so the event.get() call returns nil).

[John_ax]

thanks for the answer, I use a temp config file and work for me, with many thanks again.

echo '{"msg":"xxxx", "@timestamp":"2017-08-28T08:16:54.943Z", "service":"x.y.z", "host":"ubuntu","timestamp":1503908214}' | ./logstash   -f temp.conf