# Duplicate records in Elasticsearch

**URL:** https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658
**Category:** Logstash
**Created:** [January 20, 2021, 1:50pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658 "2021-01-20T13:50:38Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![sachin1112](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sachin1112](https://discuss.elastic.co/u/sachin1112)
#### Post date: [January 20, 2021, 1:50pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658/1 "2021-01-20T13:50:38Z")

</div>

Hello Elastic Team,

We are creating our Agile dashboards using ELK Stack , we are able to extract data and push it to ES without any issues for the first time. but when we are pushing the incremental data , we are getting the duplicate records which is affecting our visualizations. Below is the one record that we are pushing it ES  
{  
"effort": 4,  
"workitemcloseddate": null,  
"workitemstatus": "Approved",  
"workitemcreationdate": "2020-12-29T06:58:46.377Z",  
"workitemcommitteddate": null,  
"WorkitemType": "Product Backlog Item",  
"valueArea": "Business",  
"Stability": 0,  
"Plannedeffort": 113,  
"velocity": 0,  
"workitemnum": "1269729",  
"Cycletimestart": null,  
"sprint": "CICD Sprint 23",  
"sprintstartDate": "2021-01-04T00:00:00+00:00",  
"sprintfinishDate": "2021-01-14T00:00:00+00:00",  
"sprintstatus": "future",  
"teamname": "Test Team",  
"project": "ABCDemo",  
}

workitemnum and WorkitemType are unique fields and can be used to get the record in ES index . if there is any change in any of the fields like workitemstatus, workitemcloseddate on the Azure boards, then our incremental script will fetch this record but logstash will be straightaway pushing it to the ES index and the record will be duplicate( for workitemnum , WorkitemType)

Is it possible to just update the records without being get duplicated in ES ? can we handle it at logstash level or some sort of trigger job that will just update the record with the new data fields or may be drop previous record and just keep last record .

any pointers will be helpful .

Thanks,  
Sachin

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [January 20, 2021, 6:28pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658/2 "2021-01-20T18:28:12Z")

</div>

Try using a fingerprint filter to set the document id. See [this](https://discuss.elastic.co/t/threat-intelligence-avoid-redundancy-of-information-in-the-same-index/260303) thread.

---

<div class="post-metadata">

### Author: ![sachin1112](https://avatars.discourse-cdn.com/v4/letter/s/d07c76/32.png) [@sachin1112](https://discuss.elastic.co/u/sachin1112)
#### Post date: [February 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658/3 "2021-02-05T14:22:18Z")

</div>

Thanks Badger, fingerprint worked fine without any issues ..

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 5, 2021, 2:22pm UTC](https://discuss.elastic.co/t/duplicate-records-in-elasticsearch/261658/4 "2021-03-05T14:22:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.