ResourceAlreadyExists Exception After Deleting Index

[ddavid (David Dusa)]

Background

I am using python's behave library to write feature tests for my code in linux environment. Some features have data that is stored in elastic search (no clusters, 5 shards, 1 replica, version 6.4.0). For this reason, before an elastic related feature, we find a suitable open port and create a new ES instance. Before a scenario ES index is created and filled up with data. After scenario, we delete the ES index to get into a clean state.

Test runs are triggered on host using make command -j to run tests parallel. There can be 5 parallel runs at most.

Problem

The other day my feature test runs failed and from the logs the following came to me as a surprise:

**00:42:24** [2020-05-08T00:42:24,541][INFO ][o.e.c.m.MetaDataDeleteIndexService] [sw2jEeH] [sessions/d3HbZo0tSRGX-7_WHPRCUw] deleting index
...
**00:42:37** es.indices.create(index=index, body=schema["body"]) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped **00:42:37** return func(*args, params=params, **kwargs) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/client/indices.py", line 107, in create **00:42:37** params=params, body=body) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/transport.py", line 312, in perform_request **00:42:37** status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request **00:42:37** self._raise_error(response.status, raw_data) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error **00:42:37** raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) **00:42:37** elasticsearch.exceptions.RequestError: TransportError(400, 'resource_already_exists_exception', 'index [sessions/d3HbZo0tSRGX-7_WHPRCUw] already exists') **00:42:37** **00:42:37** exception: TransportError(400, 'resource_already_exists_exception', 'index [sessions/d3HbZo0tSRGX-7_WHPRCUw] already exists')

Notice that sessions index with d3HbZo0tSRGX-7_WHPRCUw uuid is stated by MetaDataDeleteIndexService as it is going to be deleted. Several seconds later, when creating a new index for an ES scenario fails like the index were existing.

My question is whether there can be a significant gap between 'deleting index' and the index is actually deleted? That would explain why ResourceAlreadyExists was thrown.

Thanking you in advance for any help and idea.

[DavidTurner]

Are you sure the deletion of the index really completed before you tried to create it again? You must wait for a successful response from the delete-index request before you can know that the index was really deleted.

[ddavid (David Dusa)]

Thank you for your quick reply!

I do not check the success of the deletion (since I am invoking es.indices.delete() with ignore=[400,404]), but it is definitely a good approach that enhances code safety.

I'll add this check and see if the problem arises again.

[DavidTurner]

Seems unwise to me to ignore 400 responses to the delete-index request. A 404 is legitimate, it means the index wasn't there, but a 400 means the request is malformed or otherwise failed.