Reduce duplicate or similar log entries

I would like to setup a logstash configuation which acts similar to the rsyslog repeated message reduction. Basically, based on parsing I would like to ship to elasticsearch only one message stating that in the previous interval, a certain amount of identical or similar log entries were seen. So I would only like to ship the summary line, not every parsed line.

There are various filters which come close, but I cannot seem to combine them in order to get what I'm looking for.

Throttle makes it possible to drop or reduce the amount of log entries based on some previous parsing, but I cannot seem to get it generate a message to be shipped like "message seen 10x in the last 60 seconds".

Counter makes it possible to count events, but again, I don't see how to ship it to elasticsearch as a regular log entry on the event that an interval threshold was expired.

Does anyone have an idea how to tackle this?

What about

@warkolm Do you have a example as to how you might make use the collate plugin for this particular problem? I am working on a very similar thing right now.

No sorry. Feel free to make a new thread with any other questions or problems though :slight_smile: