Logstash "Age" Filter Plugin

Elastic Stack Logstash
1

I am hoping to get some help getting the age plugin to work. It seems like a really new filter plugin without a log of documentation. - https://www.elastic.co/guide/en/logstash/current/plugins-filters-age.html

Here is my code

age {}
if [Timestamp][age] > 259200 {
    drop {}
}

I am trying to drop log events older than 3 days old. "Timestamp" is a grok parsed timestamp field from the incoming log in ISO8601 format. I have not been able to get the .conf to load without configuration errors. I know it works without the "age" code.

Errors

[2017-05-04T11:41:38,422][ERROR][logstash.pipeline        ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<NoMethodError: undefined method `>' for nil:NilClass>, "backtrace"=>["(eval):824:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):822:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):434:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:378:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:201:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:200:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:377:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `start_workers'"]}

[2017-05-04T11:41:38,570][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `>' for nil:NilClass>, :backtrace=>["(eval):824:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):822:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):434:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:378:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:201:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:200:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:377:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:331:in `start_workers'"]}

error1.PNG1902×82 7.32 KB

error2.PNG1902×89 6.57 KB

It seems like some sort of Ruby error. I have the Ruby plugin installed but its not part of my .conf file. The Age documentation does not specify any ruby dependencies that need to be included in filter.

Any help greatly appreciate! Thanks, Ben

2

Please don't paste screenshots. They're just too hard to read. Please include the actual text of the error between lines of triple back-ticks, like this:

```
PASTE ERRORS HERE
```

We'll see the code in a much more readable fashion.

3

my bad! fixed!

4

By default the age filter stores the age of the event in the [@metadata][age] field. If you want it in [Timestamp][age] you need to configure the age filter accordingly.

1 Like
5

Hmm what do you mean by configure the age filter accordingly? Does that mean the "Timestamp" variable needs to be defined in there not in a grok filter? I was confused by what @metadata was referring to. [@metadata][age] refer to two different variables being compared right? age being the variable storing the current time, metadata being the variable storing the time you want to compare to?

6

Hmm what do you mean by configure the age filter accordingly?

[Timestamp][age] and [@metadata][age] are just two different names of fields. The age filter by default writes to [@metadata][age]. Your conditional must read from the same field that the age filter writes to.

7

Thank you for help this worked! Now we are running into some issues installing "Age" plugin on a different environment but that is a separate issue.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.