When you submit a delete by query request, Elasticsearch gets a snapshot of the data stream or index when it begins processing the request and deletes matching documents using internal versioning. If a document changes between the time that the snapshot is taken and the delete operation is processed, it results in a version conflict and the delete operation fails.
This is what documentation says.
What I have observed is that we get a conflict error for deleteByQuery whenever there is discrepancy in the state of document in the lucene segment and the in-memory buffer.
## CASE 1: We get an version conflict when we run a DeleteByQuery on a doc whose newer version is present buffer and older version exists in the segment
# Create index with 2 shards
PUT /my_index
{
"settings": {
"number_of_shards": 8,
"refresh_interval": "60s",
"index.gc_deletes": "120s"
}
}
# Index initial document
PUT /my_index/_doc/doc1?routing=shard1
{
"entityId": "doc1",
"sequenceId": 0,
"content": "initial content"
}
# Force refresh
POST /my_index/_refresh
# Verify final state
GET /my_index/_search
{
"query": {
"term": {
"entityId": "doc1"
}
}
}
PUT /my_index/_doc/doc1?routing=shard1
{
"entityId": "doc1",
"sequenceId": 1,
"content": "initial content with PUT"
}
# 2. Event2: Delete from shard1 and index in shard2
POST /my_index/_delete_by_query?routing=shard1&wait_for_completion=true
{
"query": {
"bool": {
"must": [
{ "term": { "entityId": "doc1" } },
{ "range": { "sequenceId": { "lt": 3 } } }
]
}
}
}
These are the commands I am executing. Note that the refresh interval is 60s. I am trying to delete a doc whose version is different in lucene index and in memory buffer.
I am getting below error on deleteByQuery
{
"took": 8,
"timed_out": false,
"total": 1,
"deleted": 0,
"batches": 1,
"version_conflicts": 1,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0,
"failures": [
{
"index": "my_index",
"id": "doc1",
"cause": {
"type": "version_conflict_engine_exception",
"reason": "[doc1]: version conflict, required seqNo [0], primary term [1]. current document has seqNo [1] and primary term [1]",
"index": "my_index",
"shard": "3",
"index_uuid": "s43aw0z7Squ4hrsuOyqxFQ"
},
"status": 409
}
]
}