ILM questions2

[dennys_84 (dennys)]

Hi all. Can someone explain how ILM works? The options are pretty staight-forward but there are some questions:

1. Do we need ILM if we have a small installation?
2. We need to store our logs about 2 years, than autoremove it.
   What if we just keep that indices open for that time? Skip all phases except deletion. How it will impact the performance?
3. We configured ILM policy to rollover after 50Mb/10000 docs/7days. New index created after 10000 docs, got hot phase, the old one got warm, but logs continues writing into old, while new stays empty. How could it be, if warm should be read only?

[rugenl (Len Rugen)]

Mostly opinion answers

1. Obviously, Elasticsearch existed for a long time before ILM, so it's not required. Properly configured, it does help avoid problems like too many shards, huge shards etc.
2. Given enough ram and disk nodes are still limited by heap to total size of open indices they can support. I've seen recommendations from 5-7G, in our environment it's about 9-9.5G for our cold nodes. You can freeze indices to store more on disk with much less heap usage, but they search slower (and IMHO don't allow Kibana to search them).
3. It sounds like you have something mis-configured for that to happen. Check KIbana for ILM errors. Check to see if the first index still has the alias with is_write_alias = true. Look at the configurations of things writing to those indices.

Before ILM, curator was the common tool for lifecycle management.

[dennys_84 (dennys)]

Thank you for clearing things up)