# Avoid Duplicate document

**URL:** https://discuss.elastic.co/t/avoid-duplicate-document/84803
**Category:** Elasticsearch
**Created:** [May 6, 2017, 4:37am UTC](https://discuss.elastic.co/t/avoid-duplicate-document/84803 "2017-05-06T04:37:34Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![laspal](https://avatars.discourse-cdn.com/v4/letter/l/c6cbf5/32.png) [@laspal](https://discuss.elastic.co/u/laspal)
#### Post date: [May 6, 2017, 4:37am UTC](https://discuss.elastic.co/t/avoid-duplicate-document/84803/1 "2017-05-06T04:37:35Z")

</div>

Hi,  
I am using bulk operation to create document. The issue is its duplicate document.

> ```
> actions = [{
> "_index": index_name,
> "_type": doc_type,
> "_source": doc
> } for doc in json_list]
> try:
> helpers.bulk(esClient, actions)
> except Exception as ex:
> print("Unable to do bulk create {0}".format(index_name))
> 
> ```

json\_list1 = [{'created\_on' : 'date1', 'hostaname': 'abcd', 'msg': 'abcdef'}, {'created\_on' : 'date2', 'hostaname': 'abcdef', 'msg': 'abcdeeff'}]

json\_list2 = [{'created\_on' : 'date1', 'hostaname': 'abcd', 'msg': 'abcdef'}]

When I use json\_list2, it create new document, total - 3, whereas Thr should be only 2 document.

---

<div class="post-metadata">

### Author: ![Nilabhsagar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nilabhsagar/32/10647_2.png) [@Nilabhsagar](https://discuss.elastic.co/u/Nilabhsagar)
#### Post date: [May 6, 2017, 9:20am UTC](https://discuss.elastic.co/t/avoid-duplicate-document/84803/2 "2017-05-06T09:20:05Z")

</div>

In your code `"_id"` is missing. This should point to one of the field in your document which identify a doc as unique. The final code be like below:

```
actions = [{
                "_index": index_name,
                "_type": doc_type,
                "_source": doc,
                "_id": <A field from your doc>
            } for doc in json_list]
            try:
                helpers.bulk(esClient, actions)
            except Exception as ex:
                print("Unable to do bulk create {0}".format(index_name))
```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 3, 2017, 9:34am UTC](https://discuss.elastic.co/t/avoid-duplicate-document/84803/3 "2017-06-03T09:34:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.