Filebeat merge input duplicate logs

[dangge]

we had meet some linux filesystem error recently,a troublesome thing is that Linux will log a large number of the same logs at a certain point in time.like this:

we use filebeat and logstash to record this error and send email alert,but how can I merge these same log to single record? these repeated alert mail have filled our mailbox.
or can someone give improve suggestions? thanks for any replies.

[stephenb (Stephen Brown)]

Well If you use a Kibana alert and only set it to notify on status change you will only get one email until the condition clears then you'll get another email saying it's clear.

That is probably how I would approach the alerting side of it.

Kibana Alerting

Alerting | Kibana Guide [8.11] | Elastic

You could probably use a log threshold alert

Create a logs threshold rule | Observability Guide [7.14] | Elastic

And notify only on status change

Create a logs threshold rule | Observability Guide [7.14] | Elastic

Only on status change: Actions are not repeated when an alert remains active across checks. Actions run only when the alert status changes.

[dangge]

thanks for your suggestion, we will test it.