We started to work with ES2.4 in our project to index the content of a xml cache. This cache has an eviction process (that triggers every 30s) that removes expired documents from the index (thousands of them). To avoid using delete by query, we created an structure as follows:
- if the cache time is 1 hour we have 4 indexes, each one contains 15min of documents.
- After every 15min, a new index is created and the oldest is removed.
This was done in this way to avoid the massive document deletion that could provoke many segment merging, at least when we talk to you some years ago ...
Now with ES 6.4, is it recommended to follow with this design, or we can simply apply the delete by query to one index even if we are removing thousands of documents every 30s?