# Concurrent delete\_by\_query and indexing

**URL:** <https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868>\
**Category:** Elasticsearch\
**Created:** [December 13, 2016, 12:39pm UTC](https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868 "2016-12-13T12:39:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Serg\_Pilipenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/serg_pilipenko/32/13833_2.png) [@Serg\_Pilipenko](https://discuss.elastic.co/u/Serg_Pilipenko)\
**Post date:** [December 13, 2016, 12:39pm UTC](https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868/1 "2016-12-13T12:39:23Z")

</div>

Hi there,

I am using delete\_by\_query plugin and it's work is not quite predictable.

Here is step by step of what I do:

There is a doc in index for a long while {id: 1 and \_sync\_timestamp: 1}  
Then I concurrently run the following actions:

1. I index the updated doc  
index {document with id: 1 and \_sync\_timestamp: 2}
2. I run delete by query  
{must id: 1, must not timstamp gte: 2}

The idea behind is simple: since delete by query is asynchronous it may delete just updated docs, that's why I use \_sync\_ts + must\_not gte on \_sync\_ts.

The problem is that sometimes(very frequently) delete\_by\_query returns number of failed to delete \> 0. I don't know the reason since I am not familiar with logic the under the hood.

As far as I understand there few possible scenarios, please correct me if I am wrong:

1. if updated doc has been just indexed and merged delete\_by\_query just skips it because of \_sync\_ts
2. doc is indexed but not merged(old doc is marked as deleted, new doc is waiting to appear in index) delete\_by\_query fails on this doc
3. doc is not acknowledged at all: delete\_by\_query marks doc with \_sync\_ts as deleted and then new doc just indexed separately, no merge.

Am I right that delete\_by\_query fails on (2) scenario? Is there a better way to do the same, but w/o dealing with docs failed to delete?

Thanks,  
Sergii

---

<div class="post-metadata">

**Author:** ![cbuescher](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cbuescher/32/60402_2.png) [@cbuescher](https://discuss.elastic.co/u/cbuescher)\
**Post date:** [December 13, 2016, 5:12pm UTC](https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868/2 "2016-12-13T17:12:15Z")

</div>

Hi,

First a question, I'm confused about what you mean by "id". Is this the elasticsearch "\_id" document id or some field you are setting exlicitely? In the first case, you can simply update the document without having to delete the old one, that should be done automatically.

Regarding the second part of your question: `_delete_by_query` gets a snapshot of the index when it starts and deletes what it finds using internal versioning. That means it is not concerned with marking or merging documents on the lucene shard level. That means, if the document changes between the time when the snapshot was taken and when the delete request is processed, you get a version conflict.

---

<div class="post-metadata">

**Author:** ![Serg\_Pilipenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/serg_pilipenko/32/13833_2.png) [@Serg\_Pilipenko](https://discuss.elastic.co/u/Serg_Pilipenko)\
**Post date:** [December 13, 2016, 9:07pm UTC](https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868/3 "2016-12-13T21:07:32Z")

</div>

sorry for confusing definitions. `id` doesn't mean Elastic's `_id`. it's possible to have few docs in ES with the same `id`.

In my application I want ES is to be in sync with DB, I have `sync` function for this. `sync` is being run in queue, and after `sync(my_id)` cal doc may change state from `needs to be indexed` to `needs to be deleted`.

`my_id` represents one object in application, but may be represented by a few objects in ES. The cheapest way to perform sync of such domain is to index all docs we can find in db by `my_id`, and delete all `old` docs matching `my_id` criteria.

> That means, if the document changes between the time when the snapshot was taken and when the delete request is processed, you get a version conflict.

What is the best practice to deal with such conflict? Perform deletion, if there are failed docs wait and perform one more?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2017, 9:07pm UTC](https://discuss.elastic.co/t/concurrent-delete-by-query-and-indexing/68868/4 "2017-01-10T21:07:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.