Atomic delete and initialize index in a high concurrency environment

Elastic Stack Elasticsearch
1

I am having some problems implementing a feature in ES 2.4 that deletes and recreates an index that is currently in use, causing all kinds of weird race conditions and leaving the index in a corrupt and unusable state.

As a test environment I have a SOAP UI test suite that adds documents to the same index with random data constantly with 8 threads.

During this I call DELETE on the index which starts the delete process but fails halfway, probably because one of the concurrent ADD requests already started recreating the index. This leaves the index in an invalid state, giving an ElasticSearchException: "analyzer [western_europe] not found for field [sections]"

Preferably, I would like to be able to call the following:

  1. Tell the ES cluster to put index A in offline/maintenance mode
  2. Wait for all existing queries (read/write) to finish
  3. Delete the index
  4. Recreate the index with potential different mappings
  5. Bring the index online again.

I tried surrounding the DELETE and initialize with Open/close operations, but this does not seem to help.

Using the delete-by-query plugin for 2.4 (native supported in 5.0) is not an option, since I might need to change the mapping of existing fields in this operation which is not supported by ES and requiring a recreation of the index.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.