# Duplicate insertions for a single document

**URL:** <https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088>\
**Category:** Elasticsearch\
**Tags:** es-hadoop\
**Created:** [October 3, 2016, 3:53pm UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088 "2016-10-03T15:53:04Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\
**Post date:** [October 3, 2016, 3:53pm UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088/1 "2016-10-03T15:53:04Z")

</div>

I have a Cascading job that is reading a file of Json docs from S3 (1.5GB) and inserting it into Elasticsearch. I have 5888755 records in the file, but when I do a count in the elasticsearch index it is 6213755 ( this number varies by 100/200k records each time I run it ). The job completes correctly there are no errors on the Hadoop or Elasticsearch logs and the ES hadoop counters all have the right value. So I am not clear where the duplicates are coming from, but it looks like a bulk insert is being run more than once. I have seen the same documents appear up to 9 times. The strange thing is when I split the 1.5GB file into 5 files on and use one core EMR node the count is correct every time, but if I use 2 core nodes the count is off.

**ES Hadoop Counters**

```
        "org.elasticsearch.hadoop.mr.Counter": {
           "SCROLL_TOTAL": 0,
           "BYTES_ACCEPTED": 1747584120,
           "BYTES_RETRIED": 0,
           "DOCS_RETRIED": 0,
           "BYTES_RECEIVED": 4752000,
           "DOCS_RECEIVED": 0,
           "BULK_TOTAL_TIME_MS": 2093520,
           "BULK_RETRIES": 0,
           "DOCS_ACCEPTED": 5888755,
           "NET_RETRIES": 0,
           "BULK_TOTAL": 1188,
           "NET_TOTAL_TIME_MS": 2096311,
           "SCROLL_TOTAL_TIME_MS": 0,
           "BULK_RETRIES_TOTAL_TIME_MS": 0,
           "NODE_RETRIES": 0,
           "DOCS_SENT": 5888755,
           "BYTES_SENT": 1747584120

        },

```

**Versions**  
ES Version 2.2.0  
ES Hadoop Version: 2.2.0  
Hadoop 2.7.0

---

<div class="post-metadata">

**Author:** ![james.baiera](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james.baiera/32/10209_2.png) [@james.baiera](https://discuss.elastic.co/u/james.baiera)\
**Post date:** [October 5, 2016, 3:44pm UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088/2 "2016-10-05T15:44:57Z")

</div>

I'm assuming that you are using Amazon's EMR for the execution (or something similar)? This normally occurs when the MR environment has speculative execution turned on. The best route to get around this is to either turn off speculative execution, or if that is not possible, to assign a stable ID to each record that you are inserting (either a hash digest of the record, a composition of fields, etc.).

For more information on speculative execution woes take a look at [this section of the docs](https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration-runtime.html).

---

<div class="post-metadata">

**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\
**Post date:** [October 5, 2016, 4:52pm UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088/3 "2016-10-05T16:52:22Z")

</div>

Hi thanks for getting back to me. I am using EMR and I did have speculative execution turned on, I turned it off and it works fine. Also is it not strange that the DOCS\_ACCEPTED counter is less than the index count? Surely this is a bug as Hadoop has sent more then that number?

---

<div class="post-metadata">

**Author:** ![Pat\_Humphreys](https://avatars.discourse-cdn.com/v4/letter/p/ccd318/32.png) [@Pat\_Humphreys](https://discuss.elastic.co/u/Pat_Humphreys)\
**Post date:** [October 6, 2016, 8:53am UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088/4 "2016-10-06T08:53:30Z")

</div>

Just another follow up for anyone who has the same problem those MR properties specified in the link above have now been deprecated and you should use these in newer versions of MR:  
mapred.map.tasks.speculative.execution -\> mapreduce.map.speculative  
mapred.reduce.tasks.speculative.execution -\> mapreduce.reduce.speculative  
Have a look here  
[https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/DeprecatedProperties.html](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/DeprecatedProperties.html)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:22pm UTC](https://discuss.elastic.co/t/duplicate-insertions-for-a-single-document/62088/5 "2017-07-06T13:22:59Z")

</div>