# Detect filebeat retries to remove duplicates in the server side

**URL:** <https://discuss.elastic.co/t/detect-filebeat-retries-to-remove-duplicates-in-the-server-side/47754>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 19, 2016, 7:17am UTC](https://discuss.elastic.co/t/detect-filebeat-retries-to-remove-duplicates-in-the-server-side/47754 "2016-04-19T07:17:03Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 19, 2016, 12:35pm UTC](https://discuss.elastic.co/t/detect-filebeat-retries-to-remove-duplicates-in-the-server-side/47754/3 "2016-04-19T12:35:37Z")

</div>

filename, offset + beat (shipper name) are a good source for deduplication. One trick (when sending to logstash) is to build an `id` based on these fields and just re-index the document. I think re-indexing in elasticsearch will mark the old entry deleted and create a new one (right, takes some disk space + CPU usage, but on compaction deleted entries are finally removed from disk). It's a very simple trick to implement deduplication.

---

_[View the full topic](https://discuss.elastic.co/t/detect-filebeat-retries-to-remove-duplicates-in-the-server-side/47754)._