# Filebeat deduplication fail to update index **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 10, 2020, 5:49am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870 "2020-05-10T05:49:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mustapha.arakji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mustapha.arakji/32/65111_2.png) [@mustapha.arakji](https://discuss.elastic.co/u/mustapha.arakji)\ **Post date:** [May 10, 2020, 5:49am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/1 "2020-05-10T05:49:45Z")
Hi, I'm reading this guide: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-deduplication.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-deduplication.html), that discusses the deduplication in both filebeat and logstash, and recommends to set the document id before sending it to elasticsearch, this way the document get updated in case it already exists. But I've tried this, and I used Filebeat fingerprint processor, filebeat fails to update the document in elasticsearch if it already have the same id, while logstash does update the document successfully. I've read some comments in the discussion forms here about using logstash for such purpose as filebeat is not designed for this, but that's not what the elastic guide says! Any ideas? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 10, 2020, 9:58am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/2 "2020-05-10T09:58:48Z")
What does your config look like? Can you show two documents that should have been deduplicated but we’re not? ---
**Author:** ![mustapha.arakji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mustapha.arakji/32/65111_2.png) [@mustapha.arakji](https://discuss.elastic.co/u/mustapha.arakji)\ **Post date:** [May 11, 2020, 12:23am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/3 "2020-05-11T00:23:28Z")
What I understand, if two documents have the same "\_id" then these are considered duplicates, the last document inserted should overwrite the previous one. I'm running things in my lab for now, the configuration is very simple, reading a .csv file, you can't get it wrong, filebeat is inserting docs successfully, no issues, it's just when I insert a new line with same "myid", but slightly different info in other fields, it fails to update. ```auto processors: - fingerprint: fields: ["myId"] target_field: "@metadata._id" output.elasticsearch: hosts: ["http://localhost:9200"] index: my_index ``` What would you suggest as a configuration? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 11, 2020, 5:11am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/4 "2020-05-11T05:11:01Z")
Can you show two events that were inserted when an update/overwrite should have occurred? How are you parsing our the field you use? ---
**Author:** ![mustapha.arakji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mustapha.arakji/32/65111_2.png) [@mustapha.arakji](https://discuss.elastic.co/u/mustapha.arakji)\ **Post date:** [May 11, 2020, 9:46am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/5 "2020-05-11T09:46:07Z")
Hi Christian, This is my CSV file where filebeat is reading data from, these two events should have updated each other. I tried to push each line by itself, to garantee the order filebeat reads and index the data: ```auto myid,firstName,lastName 1001,mustapha,elastic 1001,mustapha,arakji ``` This is part of my filebeat.yml config, again, very simple stuff, nothing complicated, just copied sample config from elastic doc. ```auto processors: - decode_csv_fields: fields: message: decoded_message.csv separator: "," ignore_missing: false overwrite_keys: true trim_leading_space: false fail_on_error: true - extract_array: field: decoded_message.csv mappings: myid: 0 firstName: 1 lastName: 2 - fingerprint: fields: ["myid"] target_field: "@metadata._id" output.elasticsearch: hosts: ["http://localhost:9200"] index: my_index ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 11, 2020, 4:34pm UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/6 "2020-05-11T16:34:33Z")
Which version of Filebeat are you using? ---
**Author:** ![mustapha.arakji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mustapha.arakji/32/65111_2.png) [@mustapha.arakji](https://discuss.elastic.co/u/mustapha.arakji)\ **Post date:** [May 15, 2020, 11:15am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/8 "2020-05-15T11:15:15Z")
filebeat version 7.6.2 (amd64) ---
**Author:** ![mustapha.arakji](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mustapha.arakji/32/65111_2.png) [@mustapha.arakji](https://discuss.elastic.co/u/mustapha.arakji)\ **Post date:** [May 21, 2020, 10:03am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/9 "2020-05-21T10:03:53Z")
Anyone? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 18, 2020, 10:04am UTC](https://discuss.elastic.co/t/filebeat-deduplication-fail-to-update-index/231870/10 "2020-06-18T10:04:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.