# How to properly bulk index while defining a custom id?

**URL:** <https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671>\
**Category:** Elasticsearch\
**Created:** [October 24, 2011, 5:57pm UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671 "2011-10-24T17:57:29Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\
**Post date:** [October 24, 2011, 5:57pm UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671/1 "2011-10-24T17:57:29Z")

</div>

I've noticed different behavior when bulk indexing using a custom  
document id versus not defining a document id. By not defining an id,  
I get the desire behavior which is all documents are indexed. If I  
attempt to define an id, only one document gets indexed as opposed to  
all the documents defined in a bulk iteration.

> <https://gist.github.com/digitalsanctum/1309649>

How do you properly index all documents in a bulk request while  
defining a custom document id?

Thanks,  
Shane

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [October 24, 2011, 10:34pm UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671/2 "2011-10-24T22:34:24Z")

</div>

What is hte failure that you get? You should also see it in the longs.

On Mon, Oct 24, 2011 at 7:57 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote:

> I've noticed different behavior when bulk indexing using a custom  
> document id versus not defining a document id. By not defining an id,  
> I get the desire behavior which is all documents are indexed. If I  
> attempt to define an id, only one document gets indexed as opposed to  
> all the documents defined in a bulk iteration.
> 
> [gist:1309649 · GitHub](https://gist.github.com/1309649)
> 
> How do you properly index all documents in a bulk request while  
> defining a custom document id?
> 
> Thanks,  
> Shane

---

<div class="post-metadata">

**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\
**Post date:** [October 24, 2011, 11:01pm UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671/3 "2011-10-24T23:01:53Z")

</div>

Thanks for the reply. I see no exceptions or errors in the logs. One  
other thing I noticed is that the counts (in elasticsearch-head) are  
something like:

docs: {  
num\_docs: 3  
max\_doc: 2196  
deleted\_docs: 2193  
}

which seems to indicate that all but one of the docs are getting  
deleted for each bulk iteration.

Any additional guidance is appreciated.

Shane

On Oct 24, 6:34 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> What is hte failure that you get? You should also see it in the longs.
> 
> On Mon, Oct 24, 2011 at 7:57 PM, Shane Witbeck [sh...@digitalsanctum.com](mailto:sh...@digitalsanctum.com)wrote:
> 
> > I've noticed different behavior when bulk indexing using a custom  
> > document id versus not defining a document id. By not defining an id,  
> > I get the desire behavior which is all documents are indexed. If I  
> > attempt to define an id, only one document gets indexed as opposed to  
> > all the documents defined in a bulk iteration.
> 
> > [gist:1309649 · GitHub](https://gist.github.com/1309649)
> 
> > How do you properly index all documents in a bulk request while  
> > defining a custom document id?
> 
> > Thanks,  
> > Shane

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [October 25, 2011, 7:19am UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671/4 "2011-10-25T07:19:42Z")

</div>

On Mon, 2011-10-24 at 16:01 -0700, Shane Witbeck wrote:

> Thanks for the reply. I see no exceptions or errors in the logs. One  
> other thing I noticed is that the counts (in elasticsearch-head) are  
> something like:
> 
> docs: {  
> num\_docs: 3  
> max\_doc: 2196  
> deleted\_docs: 2193  
> }
> 
> which seems to indicate that all but one of the docs are getting  
> deleted for each bulk iteration.

You don't provide the code for getPosts or getPostID but I would suspect  
that you are reusing the same ID over and over again.

Check the \_version of the single doc that you manage to index - I bet it  
is high (when it should be 1)

clint

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:51am UTC](https://discuss.elastic.co/t/how-to-properly-bulk-index-while-defining-a-custom-id/5671/5 "2017-07-06T03:51:02Z")

</div>