Is there an automated solution for the rollover pattern?

[SphiaTower]

Hi there,

As is described in this post, the rollover pattern is very effective and can often boost performance if implemented correctly. However, implementing the rollover pattern involves too many steps, some of which are very likely to fail, especially for large indices.

The steps includes at least:

1. create an index pattern for active indices and an alias for the latest active index
2. create an index pattern for inactive indices
3. create an alias for search
4. call the rollover API and set configuration
   And after every rollover
5. set the index read-only and route all shards to a single node
6. shrink the active index to a new inactive index
7. force merge the inactive index to a single segment and flush
8. change the alias for search to include the new inactive index and exclude the old active index
9. delete the old active index
10. increase the number of replicas for the new inactive index

For a large index, some of the steps above might need 30min or even 1 hour to complete. It's difficult to monitor whether these steps succeed or not, and even more difficult to write a failover policy in case that any of the steps above has failed. Also, it's extremely easy to make mistakes if you have multiple different indices all implementing the rollover pattern.

I notice that this post was written in 2016, so is there a mature automated solution in 2019?

Thanks!

[abdon (Abdon Pijpelink)]

Yes, there is: it's called Index Lifecycle Management (ILM) - a relatively new feature in the Elastic Stack. This blog post has a good introduction and the docs are quite extensive.

[SphiaTower]

Thank you! Had I known this, I would not have written so many so many jsons