Deal with small indices (ILM Policy)

[elk-user-0001]

Hello partners!

I have set up an Elastic Stack cluster and i am performing multiple ingestions.

Some of these ingestions are abundant, ingesting around 50 GB per day, however, others contain 500... 10000 docs... that do not store more than a few megabytes.

Given the large ingestion rates, the ilm policies work perfectly since they rotate by space in a short time, however, the small ingestions fill the cluster of small indices that only manage to rotate by date (ro 50gb or 30 days).

My question is... Given indices with lower intakes, is it better to let them only rotate by space, avoiding rotation by age?

These would not be erased until reaching 50GB but they would not create multiple very small indices that pose storage balancing problems between the different nodes (taking into account that this is based on the number of shards).

What do you recommend in cases of low intake rates?

Thanks in advance !

[stephenb (Stephen Brown)]

Hi @elk-user-0001

Yes, size based is usually a bit better for a small indices.

There is a little bit of a side effect. Suppose you only do 100 mbs a day to get to 50 gb That would be 500 days..

So I think size is still good but you might want to adjust the size to equal a reasonable retention time like 30 day blocks or something like that.

Because remember when ILM deletes an indices That will be 30 days for data.

Just something to think about.

[elk-user-0001]

Thank you so much for your help !

I'm start working with Hot-Warm-Cold arch.

¿ Wich policies ares the best options with this storage method ?

[stephenb (Stephen Brown)]

Hi @elk-user-0001

The question is a little vague and there are lots of answers. It depends on the size of the indices, how long you want to keep them, and even how fast you need the responses (although that is usually for large high volume use cases)

Create a policy and give it a try you can always adjust it later.

Or tell us what you are thinking and perhaps we will have a suggestion.