# Filebeat and Duplicate entries

**URL:** <https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 2, 2019, 10:40am UTC](https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505 "2019-08-02T10:40:01Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Garry](https://avatars.discourse-cdn.com/v4/letter/g/bb73d2/32.png) [@Garry](https://discuss.elastic.co/u/Garry)\
**Post date:** [August 2, 2019, 10:40am UTC](https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505/1 "2019-08-02T10:40:01Z")

</div>

I need my Filebeat process to identify duplicate entries and update, not recreate, based on this.  
This would be similar to the document id in logstash.

Is this available using Filebeat?

My filebeat process passes the data to a pipeline....could it be done at this point?

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [August 2, 2019, 4:25pm UTC](https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505/2 "2019-08-02T16:25:30Z")

</div>

Does this help? [https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch)

---

<div class="post-metadata">

**Author:** ![Garry](https://avatars.discourse-cdn.com/v4/letter/g/bb73d2/32.png) [@Garry](https://discuss.elastic.co/u/Garry)\
**Post date:** [August 2, 2019, 8:47pm UTC](https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505/3 "2019-08-02T20:47:31Z")

</div>

Thanks, but this solution is based on logstash.

My solution does not include logstash...it is a filebeat/pipeline process.

Garry

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 30, 2019, 8:51pm UTC](https://discuss.elastic.co/t/filebeat-and-duplicate-entries/193505/4 "2019-08-30T20:51:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.