I would like to setup a logstash configuation which acts similar to the rsyslog repeated message reduction. Basically, based on parsing I would like to ship to elasticsearch only one message stating that in the previous interval, a certain amount of identical or similar log entries were seen. So I would only like to ship the summary line, not every parsed line.
There are various filters which come close, but I cannot seem to combine them in order to get what I'm looking for.
Throttle makes it possible to drop or reduce the amount of log entries based on some previous parsing, but I cannot seem to get it generate a message to be shipped like "message seen 10x in the last 60 seconds".
Counter makes it possible to count events, but again, I don't see how to ship it to elasticsearch as a regular log entry on the event that an interval threshold was expired.
Does anyone have an idea how to tackle this?
