# Duplicate documents using Filebeat

**URL:** <https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905>\
**Category:** Beats\
**Created:** [June 24, 2016, 3:52pm UTC](https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905 "2016-06-24T15:52:53Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![vitgou](https://avatars.discourse-cdn.com/v4/letter/v/779978/32.png) [@vitgou](https://discuss.elastic.co/u/vitgou)\
**Post date:** [June 24, 2016, 3:52pm UTC](https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905/1 "2016-06-24T15:52:53Z")

</div>

Hi,

I'm using Filebeat to send data to Logstash + Elasticsearch server. I noticed that if the Logstash+Elasticsearch server is not fast enough I start to see duplicate entries in elasticsearch. Is this a bug? What options do I have to remove these duplicates?

Thanks.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 24, 2016, 9:34pm UTC](https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905/2 "2016-06-24T21:34:02Z")

</div>

When there is back-pressure Logstash will close the connection from Filebeat. So for example if a batch of 10 events was in the process of being sent and in the middle of that send operation, the connection gets closed without an acknowledgement back to Filebeat for those 10 events. Then Filebeat must resend all 10 to guarantee at-least-once semantics even if in actuality some of them were successfully sent previously.

If your problem is caused by Logstash closing the connection then you will see some indicators in your Logstash logs (look for circuit breakers). You can mitigate this problem by setting the [`congestion_threshold`](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html#plugins-inputs-beats-congestion_threshold) to a very high value (years) to disable it.

---

<div class="post-metadata">

**Author:** ![vitgou](https://avatars.discourse-cdn.com/v4/letter/v/779978/32.png) [@vitgou](https://discuss.elastic.co/u/vitgou)\
**Post date:** [June 27, 2016, 9:21am UTC](https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905/3 "2016-06-27T09:21:20Z")

</div>

Now, I understand. It is working.

Thank you for your reply and your suggestion.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2016, 3:53pm UTC](https://discuss.elastic.co/t/duplicate-documents-using-filebeat/53905/4 "2016-07-15T15:53:09Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.