# Filebeat add\_id processor mechanism

**URL:** <https://discuss.elastic.co/t/filebeat-add-id-processor-mechanism/284560>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 19, 2021, 11:13am UTC](https://discuss.elastic.co/t/filebeat-add-id-processor-mechanism/284560 "2021-09-19T11:13:36Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Musketeer7](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/musketeer7/32/94193_2.png) [@Musketeer7](https://discuss.elastic.co/u/Musketeer7)\
**Post date:** [September 19, 2021, 11:13am UTC](https://discuss.elastic.co/t/filebeat-add-id-processor-mechanism/284560/1 "2021-09-19T11:13:36Z")

</div>

Hello everyone.

I have this relatively large cluster of ES, where logs from many filebeat instances are being shipped into. There are plenty of duplicates, that we can't identify their origin for sure. And because of resource restrictions we can't use the id that add\_id processor generates as \_id for Elasticsearch.  
We added add\_id processor "as a new unique id" so that we could get an idea of whether filebeat is sending the duplicates or logstash is.  
So my question is, how does filebeat handle the resends? does it generate a new unique id with add\_id when it's shipping the line for the second time? Or it send the line second time with the same unique id?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2021, 1:14pm UTC](https://discuss.elastic.co/t/filebeat-add-id-processor-mechanism/284560/2 "2021-10-17T13:14:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.