I am using python's behave library to write feature tests for my code in linux environment. Some features have data that is stored in elastic search (no clusters, 5 shards, 1 replica, version 6.4.0). For this reason, before an elastic related feature, we find a suitable open port and create a new ES instance. Before a scenario ES index is created and filled up with data. After scenario, we delete the ES index to get into a clean state.
Test runs are triggered on host using make command -j to run tests parallel. There can be 5 parallel runs at most.
The other day my feature test runs failed and from the logs the following came to me as a surprise:
**00:42:24** [2020-05-08T00:42:24,541][INFO ][o.e.c.m.MetaDataDeleteIndexService] [sw2jEeH] [sessions/d3HbZo0tSRGX-7_WHPRCUw] deleting index
**00:42:37** es.indices.create(index=index, body=schema["body"]) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/client/utils.py", line 73, in _wrapped **00:42:37** return func(*args, params=params, **kwargs) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/client/indices.py", line 107, in create **00:42:37** params=params, body=body) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/transport.py", line 312, in perform_request **00:42:37** status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/connection/http_urllib3.py", line 128, in perform_request **00:42:37** self._raise_error(response.status, raw_data) **00:42:37** File "/usr/lib/python3/dist-packages/elasticsearch/connection/base.py", line 125, in _raise_error **00:42:37** raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) **00:42:37** elasticsearch.exceptions.RequestError: TransportError(400, 'resource_already_exists_exception', 'index [sessions/d3HbZo0tSRGX-7_WHPRCUw] already exists') **00:42:37** **00:42:37** exception: TransportError(400, 'resource_already_exists_exception', 'index [sessions/d3HbZo0tSRGX-7_WHPRCUw] already exists')
Notice that sessions index with d3HbZo0tSRGX-7_WHPRCUw uuid is stated by MetaDataDeleteIndexService as it is going to be deleted. Several seconds later, when creating a new index for an ES scenario fails like the index were existing.
My question is whether there can be a significant gap between 'deleting index' and the index is actually deleted? That would explain why ResourceAlreadyExists was thrown.
Thanking you in advance for any help and idea.