# Reasons for upsert failure in bulk request

**URL:** <https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448>\
**Category:** Elasticsearch\
**Created:** [January 8, 2018, 5:26am UTC](https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448 "2018-01-08T05:26:58Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![karthik\_s1](https://avatars.discourse-cdn.com/v4/letter/k/73ab20/32.png) [@karthik\_s1](https://discuss.elastic.co/u/karthik_s1)\
**Post date:** [January 8, 2018, 5:26am UTC](https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448/1 "2018-01-08T05:26:58Z")

</div>

I would like to know under which circumstances upserts can fail. We had been doings inserts and upserts through logstash. Many a times we have noticed that some upsert doesn't happen at the same time inserts would have gone through. We didn't notice anything in the logs related to upsert failures.

Elasticsearch version: 5.2.0

---

<div class="post-metadata">

**Author:** ![luiz.santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luiz.santos/32/24664_2.png) [@luiz.santos](https://discuss.elastic.co/u/luiz.santos)\
**Post date:** [January 9, 2018, 12:22pm UTC](https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448/2 "2018-01-09T12:22:51Z")

</div>

Hi @karthik_s1,

In between the get and indexing phases of the update, it is possible that another process might have already updated the same document. By default, the update will fail with a version conflict exception. Is it your case? If not, can you provide the exception bulk request is returning?

You can use the [retry\_on\_conflict](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html#_parameters_2) parameter to control how many times to retry the update before finally throwing an exception.

Cheers,  
LG

---

<div class="post-metadata">

**Author:** ![karthik\_s1](https://avatars.discourse-cdn.com/v4/letter/k/73ab20/32.png) [@karthik\_s1](https://discuss.elastic.co/u/karthik_s1)\
**Post date:** [January 12, 2018, 5:22am UTC](https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448/3 "2018-01-12T05:22:44Z")

</div>

@luiz.santos we have seen logs at logstash, saying, bulk request failed and its retrying. But, from those logs the actual reason wasn't clear. I have read about version conflict exception, we never saw any version conflict exception in the logs.

**Other obervations:**

We observed that memory utilization was 100% in both the data nodes during the period when we lost data.

We had set retry\_on\_conflict to 5 through logstash es output plugin.

Bulk request queue was increased to 200 from 50, the queue used to hit the 200 limit sometimes.

We upgraded es version from 5.2.0 to 5.5.x due issue with circuit breakage.

Last time this data loss occurred, we upgraded the number of cores and memory (we were having low configuration, h/w upgrade was due, anyway). After upgrading the hardware (its been a week) there had been no data loss.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 9, 2018, 5:22am UTC](https://discuss.elastic.co/t/reasons-for-upsert-failure-in-bulk-request/114448/4 "2018-02-09T05:22:45Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.