Version conflict always on _delete_from_query

Elastic Stack Elasticsearch
1

Hi,

I have a simple index. When I add document, this document has a version of 1 as shown below.

If I then call _delete_for_update .... I always get version conflict and I don't know why. If I re-run the delete script then it is deleted as expected ...

There was no other insert nor update .... I call php script for insert and delete manually ....

Here is my example :

    {
            "_index" : "i8grpusr1",
            "_type" : "grpusr",
            "_id" : "70437463654587_447",
            "_version" : 1,
            "_score" : 1.0,
            "_source" : {
              "fk_groups" : "447",
              "fullname" : "",
              "gid_users" : "70437463654587"
            }
          },
          {
            "_index" : "i8grpusr1",
            "_type" : "grpusr",
            "_id" : "70437463654587_361",
            "_version" : 1,
            "_score" : 1.0,
            "_source" : {
              "fk_groups" : "361",
              "fullname" : "",
              "gid_users" : "70437463654587"
            }
          },
          {
            "_index" : "i8grpusr1",
            "_type" : "grpusr",
            "_id" : "70437463654587_449",
            "_version" : 1,
            "_score" : 1.0,
            "_source" : {
              "fk_groups" : "449",
              "fullname" : "",
              "gid_users" : "70437463654587"
            }
          }

After I all _delete_for_update I get this :

{
	"took": 3,
	"timed_out": false,
	"total": 3,
	"deleted": 1,
	"batches": 1,
	"version_conflicts": 2,
	"noops": 0,
	"retries": {
		"bulk": 0,
		"search": 0
	},
	"throttled_millis": 0,
	"requests_per_second": -1.0,
	"throttled_until_millis": 0,
	"failures": [{
		"index": "i8grpusr1",
		"type": "grpusr",
		"id": "70437463654587_447",
		"cause": {
			"type": "version_conflict_engine_exception",
			"reason": "[grpusr][70437463654587_447]: version conflict, current version [2] is different than the one provided [1]",
			"index_uuid": "kZXuIQAiSEy7HbdDs6KHXg",
			"shard": "0",
			"index": "i8grpusr1"
		},
		"status": 409
	}, {
		"index": "i8grpusr1",
		"type": "grpusr",
		"id": "70437463654587_449",
		"cause": {
			"type": "version_conflict_engine_exception",
			"reason": "[grpusr][70437463654587_449]: version conflict, current version [2] is different than the one provided [1]",
			"index_uuid": "kZXuIQAiSEy7HbdDs6KHXg",
			"shard": "0",
			"index": "i8grpusr1"
		},
		"status": 409
	}]
}

And I call _delete_by_query like this :

{
	"query": {
		"term": {
			"fk_groups": ' . $id . '
		}
	}
}
2

May be you are updating some documents while trying to remove them?

You can use ?conflicts=proceed If you don't want to abort but just count the conflicted documents.

3

Hi,
I do bulk insert and the result is what I've showed above.
Then I do delete by query .... that's it.

Counting is not ok, cause I need to make sure all documents are deleted .....

I do insert with PUT
xxxx/grpusr/70437463654587_447?refresh&filter_path=created,result&pretty

Thank you

4

Hi,

I might have found the problem ....
I have users and groups .... user owns some groups ... and can be part of some other group

If I delete user... then I first go end delete groups he owns .... and I call _delete_by_query ... where query is by group ID ....

Then go and delete user ... and I again call _delete_by_query on the same index as before but on user ID ...

So is it possible that _delete_by_query increments version until it is deleted ?
I don't call REFRESH when deleting .... as I do when I ADD ...

And for some reason first delete didn't finish processing in ES, and cause I call it again then the version conflict appears ?

Update:

I added REFRESH while calling _Delete_for_update and it looks ok now, without conflicts ....

Question: Will adding refresh cause performance issues when there will be a few million rows ?
If yes, should we build a logic without calling refresh ? OK this would mean that user will see results after some time ... but how much time is this ? A few minutes is ok , hours is not ...
Of course we don't expect _delete_for_query to be called 100 times a second, but every 30sec it might ...
Thank you.

5

Yes. Deleting a document does increase the version. It's like an update which is marking a document to be removed eventually.

If you run both scripts at the same time, that might explain.

6

Can you please say something regarding performance that I wrote ?
thank you.

7

Calling refresh will cause indeed performance problems IMO.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.