# All IndexRequestBuilders in same thread use the same source array? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2011, 11:03pm UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192 "2011-04-05T23:03:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jacorob](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jacorob](https://discuss.elastic.co/u/jacorob)\ **Post date:** [April 5, 2011, 11:03pm UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/1 "2011-04-05T23:03:47Z")
I'm trying to process a number of IndexRequests through the Bulk API in 0.15.2 through the java api. To do so I have a loop that first creates all the IndexRequestBuilders and stores them in a list. At a later time I then iterate through the list and add each indexrequestbuilder to particular bulk request. The documents I'm indexing will all generate the exact same size source, but the content of the source will be different (different constant size guids, etc). Each individual IndexRequestBuilders is generated correctly. However, setting the source on a new IndexRequestBuilder changes the source on EVERY IndexRequestBuilder that was created before it. Looking at the IndexRequestBuilders instances, all the builders use the exact same source instance (a byte array). Is this a bug or as designed? If as designed, why? Am I mistaken in that this limits the number of IndexRequestBuilders that you can have instantiated at one time to one per thread? Thanks, Bob ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 5, 2011, 11:06pm UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/2 "2011-04-05T23:06:38Z")
This is by design in order to reduce array copies when using single index requests. When you add an index request builder to a bulk request, it gets copied over. What you need to do is keep the bulk request around and keep adding to it, instead of keeping the index builders around. On Wednesday, April 6, 2011 at 2:03 AM, Bob wrote: > I'm trying to process a number of IndexRequests through the Bulk API > in 0.15.2 through the java api. To do so I have a loop that first > creates all the IndexRequestBuilders and stores them in a list. At a > later time I then iterate through the list and add each > indexrequestbuilder to particular bulk request. The documents I'm > indexing will all generate the exact same size source, but the content > of the source will be different (different constant size guids, etc). > > Each individual IndexRequestBuilders is generated correctly. However, > setting the source on a new IndexRequestBuilder changes the source on > EVERY IndexRequestBuilder that was created before it. Looking at the > IndexRequestBuilders instances, all the builders use the exact same > source instance (a byte array). > > Is this a bug or as designed? If as designed, why? Am I mistaken in > that this limits the number of IndexRequestBuilders that you can have > instantiated at one time to one per thread? > > Thanks, > Bob ---
**Author:** ![jacorob](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jacorob](https://discuss.elastic.co/u/jacorob)\ **Post date:** [April 6, 2011, 3:59am UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/3 "2011-04-06T03:59:08Z")
Thanks, Shay. As soon as I figured out what was going on I started passing my bulk request around everywhere. Fixed things right up. If you can only realistically ever have 1 active instance of an IndexRequestBuilder per thread, why do you allow users to create multiple instances of IndexRequestBuilder in the same thread in the first place? Seems like that just asks for people to screw it up like I did. Are there use cases where one would actually want/be able to have such behavior across multiple instances? Bob On Tue, Apr 5, 2011 at 6:06 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > This is by design in order to reduce array copies when using single index > requests. When you add an index request builder to a bulk request, it gets > copied over. What you need to do is keep the bulk request around and keep > adding to it, instead of keeping the index builders around. > > On Wednesday, April 6, 2011 at 2:03 AM, Bob wrote: > > I'm trying to process a number of IndexRequests through the Bulk API > in 0.15.2 through the java api. To do so I have a loop that first > creates all the IndexRequestBuilders and stores them in a list. At a > later time I then iterate through the list and add each > indexrequestbuilder to particular bulk request. The documents I'm > indexing will all generate the exact same size source, but the content > of the source will be different (different constant size guids, etc). > > Each individual IndexRequestBuilders is generated correctly. However, > setting the source on a new IndexRequestBuilder changes the source on > EVERY IndexRequestBuilder that was created before it. Looking at the > IndexRequestBuilders instances, all the builders use the exact same > source instance (a byte array). > > Is this a bug or as designed? If as designed, why? Am I mistaken in > that this limits the number of IndexRequestBuilders that you can have > instantiated at one time to one per thread? > > Thanks, > Bob ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 6, 2011, 9:13am UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/4 "2011-04-06T09:13:53Z")
Well, its not really the index request builder, its the XContentBuilder used. By default, you an use XContentBuilder#jsonBuilder, but you can also use XContentFactory#safeJsonBuilder which will not use thread local buffer. On Wednesday, April 6, 2011 at 6:59 AM, Bob Jacoby wrote: > Thanks, Shay. As soon as I figured out what was going on I started passing my bulk request around everywhere. Fixed things right up. > > If you can only realistically ever have 1 active instance of an IndexRequestBuilder per thread, why do you allow users to create multiple instances of IndexRequestBuilder in the same thread in the first place? Seems like that just asks for people to screw it up like I did. Are there use cases where one would actually want/be able to have such behavior across multiple instances? > > Bob > > On Tue, Apr 5, 2011 at 6:06 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) wrote: > > > This is by design in order to reduce array copies when using single index requests. When you add an index request builder to a bulk request, it gets copied over. What you need to do is keep the bulk request around and keep adding to it, instead of keeping the index builders around. > > On Wednesday, April 6, 2011 at 2:03 AM, Bob wrote: > > > > > I'm trying to process a number of IndexRequests through the Bulk API > > > in 0.15.2 through the java api. To do so I have a loop that first > > > creates all the IndexRequestBuilders and stores them in a list. At a > > > later time I then iterate through the list and add each > > > indexrequestbuilder to particular bulk request. The documents I'm > > > indexing will all generate the exact same size source, but the content > > > of the source will be different (different constant size guids, etc). > > > > > > Each individual IndexRequestBuilders is generated correctly. However, > > > setting the source on a new IndexRequestBuilder changes the source on > > > EVERY IndexRequestBuilder that was created before it. Looking at the > > > IndexRequestBuilders instances, all the builders use the exact same > > > source instance (a byte array). > > > > > > Is this a bug or as designed? If as designed, why? Am I mistaken in > > > that this limits the number of IndexRequestBuilders that you can have > > > instantiated at one time to one per thread? > > > > > > Thanks, > > > Bob ---
**Author:** ![jacorob](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jacorob](https://discuss.elastic.co/u/jacorob)\ **Post date:** [April 6, 2011, 2:07pm UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/5 "2011-04-06T14:07:08Z")
Ah. Makes sense. Thanks again, Shay! I'm constantly amazed at how fast you respond to questions on this list! Bob On Wed, Apr 6, 2011 at 4:13 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Well, its not really the index request builder, its the XContentBuilder > used. By default, you an use XContentBuilder#jsonBuilder, but you can also > use XContentFactory#safeJsonBuilder which will not use thread local buffer. > > On Wednesday, April 6, 2011 at 6:59 AM, Bob Jacoby wrote: > > Thanks, Shay. As soon as I figured out what was going on I started passing > my bulk request around everywhere. Fixed things right up. > > If you can only realistically ever have 1 active instance of an > IndexRequestBuilder per thread, why do you allow users to create multiple > instances of IndexRequestBuilder in the same thread in the first place? > Seems like that just asks for people to screw it up like I did. Are there > use cases where one would actually want/be able to have such behavior across > multiple instances? > > Bob > > On Tue, Apr 5, 2011 at 6:06 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > This is by design in order to reduce array copies when using single index > requests. When you add an index request builder to a bulk request, it gets > copied over. What you need to do is keep the bulk request around and keep > adding to it, instead of keeping the index builders around. > > On Wednesday, April 6, 2011 at 2:03 AM, Bob wrote: > > I'm trying to process a number of IndexRequests through the Bulk API > in 0.15.2 through the java api. To do so I have a loop that first > creates all the IndexRequestBuilders and stores them in a list. At a > later time I then iterate through the list and add each > indexrequestbuilder to particular bulk request. The documents I'm > indexing will all generate the exact same size source, but the content > of the source will be different (different constant size guids, etc). > > Each individual IndexRequestBuilders is generated correctly. However, > setting the source on a new IndexRequestBuilder changes the source on > EVERY IndexRequestBuilder that was created before it. Looking at the > IndexRequestBuilders instances, all the builders use the exact same > source instance (a byte array). > > Is this a bug or as designed? If as designed, why? Am I mistaken in > that this limits the number of IndexRequestBuilders that you can have > instantiated at one time to one per thread? > > Thanks, > Bob ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:08am UTC](https://discuss.elastic.co/t/all-indexrequestbuilders-in-same-thread-use-the-same-source-array/4192/6 "2017-07-06T04:08:49Z")