I have single node Elasticsearch and a test case where record is created with custom _id field and gets deleted immediately by using _id (/_bulk) fails, sometimes partially works. but If I put say 2sec sleep between creation and deletion test cases passes 100%, is this expected behavior?
Hi @kam_ka,
Is this being executed as part of a unit or integration test? What error are you receiving?
It is possible it could be down to a timing issue.
No, it is is single test case that I am working with. I don't get err with bulk operation, rather get 404, not_found message for the delete by _id request for all or partial messages.
even if try with 100ms it works for the test (10 times run without failure (-count=10))
when I bring the sleep time to 10ms things start breaking:
I was wrong when I said single doc, test creates 30 doc and tries to delete immediately.
for example:
| Error: | Not equal: | |
|---|---|---|
| expected: 0 | ||
| actual : 5 |
after delete system finds 5 docs, one case 30 docs, other 8 cases it passed. After deletion run system runs search for 5 seconds and expects at the end of 5seconds no docs will be in the index, but finding 5, sometimes 30 etc
Exactly how are you deleting the documents? Are you using single delete requests? Are you song a single bulk request with multiple delete commands? Are you sending a delete by query request in which you specify the ids to delete?
this is what is sent to http://localhost:9200/doc_index/_bulk:
{"delete":{"_id":"6008017780171595263","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"3775702373204005854","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"1543386966236416445","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"17757815632978378652","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"15525500226010789243","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"13293184819043199834","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"11060869412075610425","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"8828554005108021016","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"6596238598140431607","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"4363923191172842198","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"2131607784205252789","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"18346036450947214996","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"16113721043979625587","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"13881405637012036178","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"11649090230044446769","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"9416774823076857360","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"7184459416109267951","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"4952144009141678542","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"2719828602174089133","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"487513195206499724","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"16701941861948461931","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"14469626454980872522","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"12237311048013283113","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"10004995641045693704","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"7772680234078104295","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"5540364827110514886","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"3308049420142925477","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"1075734013175336068","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"17290162679917298275","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
{"delete":{"_id":"15057847272949708866","_index":"doc_index","routing":"1c72f2ca-06e2-46ab-bef6-465704c0d0ee"}}
How many primary shards does the index you are deleting from have?
As you are using routing I would also recommend verifying that you submitted the same routing value when you indexed the documents?
it is:
"settings": {
"number_of_shards": 6,
"number_of_replicas": 1
},
It is not routing issue because if I put a sleep for 100ms after doc creation before issuing delete query test passes, this is timing issue, my understanding was deletion by _id bypasses nrt.
That is correct, which is why I was looking for other explanations and to rule out other potential causes.
I assume you are sending the delete request only after the indexing request has completed successfully and returned so there are no timing issues around completion?
I think this is the cause, basically indexing happens at the end of the queue, the same way deletion happens at the end of the queue. But multiple message processors are there which is possible that indexing didn't fully finish. I think we can close this, thanks for pointing out.
1 Like