# Filebeat merge input duplicate logs

**URL:** <https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 17, 2021, 4:56pm UTC](https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506 "2021-09-17T16:56:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dangge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dangge/32/94734_2.png) [@dangge](https://discuss.elastic.co/u/dangge)\
**Post date:** [September 17, 2021, 4:57pm UTC](https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506/1 "2021-09-17T16:57:00Z")

</div>

we had meet some linux filesystem error recently,a troublesome thing is that Linux will log a large number of the same logs at a certain point in time.like this:

 ![20210918004857](https://us1.discourse-cdn.com/elastic/original/3X/4/8/4859d958752b550470730c1d38796748b943d0d0.png)  
we use filebeat and logstash to record this error and send email alert,but how can I merge these same log to single record? these repeated alert mail have filled our mailbox.  
or can someone give improve suggestions? thanks for any replies.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [September 17, 2021, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506/2 "2021-09-17T17:43:12Z")

</div>

Well If you use a Kibana alert and only set it to notify on status change you will only get one email until the condition clears then you'll get another email saying it's clear.

That is probably how I would approach the alerting side of it.

Kibana Alerting

> **[Alerting | Kibana Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/kibana/current/alerting-getting-started.html)**

You could probably use a log threshold alert

> **[Create a logs threshold rule | Observability Guide \[7.14\] | Elastic](https://www.elastic.co/guide/en/observability/7.14/logs-threshold-alert.html)**

And notify only on status change

> **[Create a logs threshold rule | Observability Guide \[7.14\] | Elastic](https://www.elastic.co/guide/en/observability/7.14/logs-threshold-alert.html)**

> Only on status change: Actions are not repeated when an alert remains active across checks. Actions run only when the alert status changes.

---

<div class="post-metadata">

**Author:** ![dangge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dangge/32/94734_2.png) [@dangge](https://discuss.elastic.co/u/dangge)\
**Post date:** [September 18, 2021, 6:53am UTC](https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506/3 "2021-09-18T06:53:36Z")

</div>

thanks for your suggestion, we will test it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2021, 8:54am UTC](https://discuss.elastic.co/t/filebeat-merge-input-duplicate-logs/284506/4 "2021-10-16T08:54:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.