# More Index Latency with add\_id proc in Filebeat

**URL:** <https://discuss.elastic.co/t/more-index-latency-with-add-id-proc-in-filebeat/364518>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [August 7, 2024, 8:14am UTC](https://discuss.elastic.co/t/more-index-latency-with-add-id-proc-in-filebeat/364518 "2024-08-07T08:14:40Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![demudrol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/demudrol/32/24030_2.png) [@demudrol](https://discuss.elastic.co/u/demudrol)\
**Post date:** [August 7, 2024, 8:14am UTC](https://discuss.elastic.co/t/more-index-latency-with-add-id-proc-in-filebeat/364518/1 "2024-08-07T08:14:40Z")

</div>

Hi! We have logging structure like with:  
Application -\> docker json -\> Filebeat file harvester -\> Kafka Topic -\> Filebeat retranslator from Topic to Elasticsearch.  
Sometimes there is a lot of event deduplication in Elasticsearch. According to this article [Deduplicate data | Filebeat Reference [7.17] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-deduplication.html) add\_id processor might help.  
Yesterday applied with configuration and saw that Index Latency now higher in 2 or 3 times.  
I though that if filebeat make metadata\_id, latency must be even less cause less work on Elasticsearch but it don't. Could explain why?

 ![2024-08-07 11.14.15](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a6dbe142553c41402a48a0f25ba370d4446308e.jpeg)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [August 7, 2024, 8:21am UTC](https://discuss.elastic.co/t/more-index-latency-with-add-id-proc-in-filebeat/364518/2 "2024-08-07T08:21:58Z")

</div>

When you specify a document ID outside of Elasticsearch, e.g. in Filebeat, every index operation is essentially a potential update and more expensive than if Elasticsearch is allowed to set the document ID (it knows the ID is uniwu and can just insert it). This is the price you pay to avoid/limit duplicates.