# VersionConflictEngineException in 0.19.1 when updating via mapper attachments plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2012, 8:21pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132 "2012-03-26T20:21:41Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 26, 2012, 8:21pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/1 "2012-03-26T20:21:41Z")
I'm using a couple of workers to update existing documents with one or more attachment/files using a worker pattern. 99% of the time this works but once in a while I'm getting the following exception: > The update code is here: > Any ideas why the exception would occur? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 27, 2012, 1:22pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/2 "2012-03-27T13:22:33Z")
The way update works is by reading the document from the shard, and then indexing it (using the version it was read with). There might be a version conflict happening if two updates on the same doc happen at the same time, and then interleave. You can set the retryOnConflict to a higher value to automatically retry the update if it happens. On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > I'm using a couple of workers to update existing documents with one or > more attachment/files using a worker pattern. 99% of the time this works > but once in a while I'm getting the following exception: > > [ES 0.19.1 VersionConflictEngineException · GitHub](https://gist.github.com/2209367) > > The update code is here: > > [ES update · GitHub](https://gist.github.com/2209383) > > Any ideas why the exception would occur? ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 27, 2012, 1:39pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/3 "2012-03-27T13:39:15Z")
That makes sense if I have two or more processes updating the same doc. Is there a way to handle this more gracefully other than increasing the retryOnConflict? Otherwise, I'll have to synchronize my processes to only do one update at a time for a document. Thanks, Shane On Tuesday, March 27, 2012 9:22:33 AM UTC-4, kimchy wrote: > The way update works is by reading the document from the shard, and then > indexing it (using the version it was read with). There might be a version > conflict happening if two updates on the same doc happen at the same time, > and then interleave. You can set the retryOnConflict to a higher value to > automatically retry the update if it happens. > > On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > I'm using a couple of workers to update existing documents with one or > > more attachment/files using a worker pattern. 99% of the time this works > > but once in a while I'm getting the following exception: > > > > [ES 0.19.1 VersionConflictEngineException · GitHub](https://gist.github.com/2209367) > > > > The update code is here: > > > > [ES update · GitHub](https://gist.github.com/2209383) > > > > Any ideas why the exception would occur? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 27, 2012, 2:30pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/4 "2012-03-27T14:30:51Z")
What do you mean by handling it more gracefully? What do you have in mind? The main idea here is not to block the indexing process while the update is happening. You can have a high retry on conflict value... On Tue, Mar 27, 2012 at 3:39 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > That makes sense if I have two or more processes updating the same doc. Is > there a way to handle this more gracefully other than increasing the > retryOnConflict? Otherwise, I'll have to synchronize my processes to only > do one update at a time for a document. > > Thanks, > Shane > > On Tuesday, March 27, 2012 9:22:33 AM UTC-4, kimchy wrote: > > > The way update works is by reading the document from the shard, and then > > indexing it (using the version it was read with). There might be a version > > conflict happening if two updates on the same doc happen at the same time, > > and then interleave. You can set the retryOnConflict to a higher value to > > automatically retry the update if it happens. > > > > On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck \<[shane@digitalsanctum.com](mailto:shane@digitalsanctum.com) > > > > > wrote: > > > > > I'm using a couple of workers to update existing documents with one or > > > more attachment/files using a worker pattern. 99% of the time this works > > > but once in a while I'm getting the following exception: > > > > > > [https://gist.github.com/\*\*2209367](https://gist.github.com/**2209367) [https://gist.github.com/2209367](https://gist.github.com/2209367) > > > > > > The update code is here: > > > > > > [https://gist.github.com/\*\*2209383](https://gist.github.com/**2209383) [https://gist.github.com/2209383](https://gist.github.com/2209383) > > > > > > Any ideas why the exception would occur? ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 27, 2012, 2:45pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/5 "2012-03-27T14:45:58Z")
For background, I have documents that are currently indexed in two phases. 1. First phase is fairly quick and indexes all basic data. 2. Second phase is potentially longer running since it's using the attachment-mapper plugin to update the document with possibly several attachments ranging in size from a few KB to several MB's. The idea here to get all documents indexed as quickly as possible with basic data then have several workers index the related attachments. While I understand your design decision to not block the indexing process, I think this should be mentioned in the documentation somewhere. I'll try your suggestion of increasing the retryOnConflict but it seems better in my case to manually synchronize the second phase to minimize indexing time and resources. Would you agree this is the better approach? Thanks, Shane On Tuesday, March 27, 2012 10:30:51 AM UTC-4, kimchy wrote: > What do you mean by handling it more gracefully? What do you have in mind? > The main idea here is not to block the indexing process while the update is > happening. You can have a high retry on conflict value... > > On Tue, Mar 27, 2012 at 3:39 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > That makes sense if I have two or more processes updating the same doc. > > Is there a way to handle this more gracefully other than increasing the > > retryOnConflict? Otherwise, I'll have to synchronize my processes to only > > do one update at a time for a document. > > > > Thanks, > > Shane > > > > On Tuesday, March 27, 2012 9:22:33 AM UTC-4, kimchy wrote: > > > > > The way update works is by reading the document from the shard, and then > > > indexing it (using the version it was read with). There might be a version > > > conflict happening if two updates on the same doc happen at the same time, > > > and then interleave. You can set the retryOnConflict to a higher value to > > > automatically retry the update if it happens. > > > > > > On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck \< > > > [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)\> wrote: > > > > > > > I'm using a couple of workers to update existing documents with one or > > > > more attachment/files using a worker pattern. 99% of the time this works > > > > but once in a while I'm getting the following exception: > > > > > > > > [https://gist.github.com/\*\*2209367](https://gist.github.com/**2209367) [https://gist.github.com/2209367](https://gist.github.com/2209367) > > > > > > > > The update code is here: > > > > > > > > [https://gist.github.com/\*\*2209383](https://gist.github.com/**2209383) [https://gist.github.com/2209383](https://gist.github.com/2209383) > > > > > > > > Any ideas why the exception would occur? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 27, 2012, 6:50pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/6 "2012-03-27T18:50:36Z")
Not really sure I understand..., are you indexing a doc, and then adding attachments to it using update API? Why not index the whole doc with all the attachments at once? On Tue, Mar 27, 2012 at 4:45 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > For background, I have documents that are currently indexed in two phases. > > 1. First phase is fairly quick and indexes all basic data. > 2. Second phase is potentially longer running since it's using the > attachment-mapper plugin to update the document with possibly several > attachments ranging in size from a few KB to several MB's. > > The idea here to get all documents indexed as quickly as possible with > basic data then have several workers index the related attachments. While I > understand your design decision to not block the indexing process, I think > this should be mentioned in the documentation somewhere. I'll try your > suggestion of increasing the retryOnConflict but it seems better in my case > to manually synchronize the second phase to minimize indexing time and > resources. Would you agree this is the better approach? > > Thanks, > > Shane > > On Tuesday, March 27, 2012 10:30:51 AM UTC-4, kimchy wrote: > > > What do you mean by handling it more gracefully? What do you have in > > mind? The main idea here is not to block the indexing process while the > > update is happening. You can have a high retry on conflict value... > > > > On Tue, Mar 27, 2012 at 3:39 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > > > That makes sense if I have two or more processes updating the same doc. > > > Is there a way to handle this more gracefully other than increasing the > > > retryOnConflict? Otherwise, I'll have to synchronize my processes to only > > > do one update at a time for a document. > > > > > > Thanks, > > > Shane > > > > > > On Tuesday, March 27, 2012 9:22:33 AM UTC-4, kimchy wrote: > > > > > > > The way update works is by reading the document from the shard, and > > > > then indexing it (using the version it was read with). There might be a > > > > version conflict happening if two updates on the same doc happen at the > > > > same time, and then interleave. You can set the retryOnConflict to a higher > > > > value to automatically retry the update if it happens. > > > > > > > > On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck \< > > > > [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)\> wrote: > > > > > > > > > I'm using a couple of workers to update existing documents with one or > > > > > more attachment/files using a worker pattern. 99% of the time this works > > > > > but once in a while I'm getting the following exception: > > > > > > > > > > [https://gist.github.com/\*\*220936\*\*7](https://gist.github.com/ **220936** 7) [https://gist.github.com/2209367](https://gist.github.com/2209367) > > > > > > > > > > The update code is here: > > > > > > > > > > [https://gist.github.com/\*\*220938\*\*3](https://gist.github.com/ **220938** 3) [https://gist.github.com/2209383](https://gist.github.com/2209383) > > > > > > > > > > Any ideas why the exception would occur? ---
**Author:** ![Shane\_Witbeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shane_witbeck/32/2803_2.png) [@Shane\_Witbeck](https://discuss.elastic.co/u/Shane_Witbeck)\ **Post date:** [March 27, 2012, 7:00pm UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/7 "2012-03-27T19:00:23Z")
Correct. I'm indexing a doc, then adding attachments via update API. I'm doing a two-phase approach to get the basic information indexed more quickly instead of waiting for all associated attachments to get indexed up front. This way I'm able to index over 1M documents with all data other than attachments in ~20 mins instead of waiting for ~200K attachments to get indexed which takes much longer. Anyway, I increased the retryOnConflict to 20 and the VersionConflictEngineException went away. Thanks, Shane On Tuesday, March 27, 2012 2:50:36 PM UTC-4, kimchy wrote: > Not really sure I understand..., are you indexing a doc, and then adding > attachments to it using update API? Why not index the whole doc with all > the attachments at once? > > On Tue, Mar 27, 2012 at 4:45 PM, Shane Witbeck [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)wrote: > > > For background, I have documents that are currently indexed in two > > phases. > > > > 1. First phase is fairly quick and indexes all basic data. > > 2. Second phase is potentially longer running since it's using the > > attachment-mapper plugin to update the document with possibly several > > attachments ranging in size from a few KB to several MB's. > > > > The idea here to get all documents indexed as quickly as possible with > > basic data then have several workers index the related attachments. While I > > understand your design decision to not block the indexing process, I think > > this should be mentioned in the documentation somewhere. I'll try your > > suggestion of increasing the retryOnConflict but it seems better in my case > > to manually synchronize the second phase to minimize indexing time and > > resources. Would you agree this is the better approach? > > > > Thanks, > > > > Shane > > > > On Tuesday, March 27, 2012 10:30:51 AM UTC-4, kimchy wrote: > > > > > What do you mean by handling it more gracefully? What do you have in > > > mind? The main idea here is not to block the indexing process while the > > > update is happening. You can have a high retry on conflict value... > > > > > > On Tue, Mar 27, 2012 at 3:39 PM, Shane Witbeck \<[shane@digitalsanctum.com](mailto:shane@digitalsanctum.com) > > > > > > > wrote: > > > > > > > That makes sense if I have two or more processes updating the same doc. > > > > Is there a way to handle this more gracefully other than increasing the > > > > retryOnConflict? Otherwise, I'll have to synchronize my processes to only > > > > do one update at a time for a document. > > > > > > > > Thanks, > > > > Shane > > > > > > > > On Tuesday, March 27, 2012 9:22:33 AM UTC-4, kimchy wrote: > > > > > > > > > The way update works is by reading the document from the shard, and > > > > > then indexing it (using the version it was read with). There might be a > > > > > version conflict happening if two updates on the same doc happen at the > > > > > same time, and then interleave. You can set the retryOnConflict to a higher > > > > > value to automatically retry the update if it happens. > > > > > > > > > > On Mon, Mar 26, 2012 at 10:21 PM, Shane Witbeck \< > > > > > [shane@digitalsanctum.com](mailto:shane@digitalsanctum.com)\> wrote: > > > > > > > > > > > I'm using a couple of workers to update existing documents with one > > > > > > or more attachment/files using a worker pattern. 99% of the time this works > > > > > > but once in a while I'm getting the following exception: > > > > > > > > > > > > [https://gist.github.com/\*\*220936\*\*7](https://gist.github.com/ **220936** 7) [https://gist.github.com/2209367](https://gist.github.com/2209367) > > > > > > > > > > > > The update code is here: > > > > > > > > > > > > [https://gist.github.com/\*\*220938\*\*3](https://gist.github.com/ **220938** 3) [https://gist.github.com/2209383](https://gist.github.com/2209383) > > > > > > > > > > > > Any ideas why the exception would occur? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:34am UTC](https://discuss.elastic.co/t/versionconflictengineexception-in-0-19-1-when-updating-via-mapper-attachments-plugin/7132/8 "2017-07-06T03:34:33Z")