Drop old messages using Ruby filter

Elastic Stack Logstash
1

Greetings

Occasionally, my logstash receives a message from few days back, and logstash tries to write it into an index which was already marked as readonly and forcemerged. Logstash then receives a 403 error from Elasticsearch, and instead of dropping the message or placing it into DLQ, it keeps retrying it.
Eventually, these undeliverable messages clog up the output queue completely, and no valid events are emitted afterwards.
I submitted a bug report already, but there seems to be no activity on it:

I tried using the following filter:

filter {
  ruby {
    init => "require 'time'"
    code => 'if event.get("[@timestamp]") < ( Time.now - 432000 )
      event.cancel
    end'
  }
}

However, using this, I get the following error:

[ERROR][logstash.filters.ruby    ] Ruby exception occurred: comparison of LogStash::Timestamp with Time failed

I also tried to follow this answer, but in my environment, installing additional plugins like age{} is a lengthy process requiring aprovals, so I'd prefer to avoid that:

Is there an easy way to use the Ruby filter to achieve this?
Thanks

2 Likes
2

I found a way:

ruby {
  init => "require 'time'"
  code => 'if LogStash::Timestamp.new(event.get("@timestamp")+432000) < ( LogStash::Timestamp.now)
    event.cancel
  end'
}
2 Likes
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.