Shrink using ILM

[Phandora (Mayte)]

Hi,

We are designing our cluster taking into consideration some Elasticsearch best practices, focusing on keeping the shard size as close to our desired one (currently 25gb) as possible.

We decided to create ILM policies in order to achieve that, by rolling the indices using max_size and max_age to simplify the index management. However, age rotation may lead to smaller indices.

The shrink action would be a good choice in this case, but handling it with ILM seems to have certain limitations since the only setting available is number_of_shards.

It would be more convenient to shrink the indices taking into account the index primary shards and its size since we can not know in advance which number of shards will be accurate.

Is there any feature that might allow us to achieve this? Can it be approached differently?

I would appreciate any help.

Best regards.

[rugenl (Len Rugen)]

It's pretty much a trade off between retention granularity and shard size. Say I want to retain data 6 months, if I set rollover age to 15 days, some data will get the extra 15 days, some won't, assuming the rollover occurs because of age. It's still better than daily indices.

Since we are multi-tenant, IMHO, tuning for cluster performance instead of individual index performance, I can almost always use 1 shard (and 1 replica). Since many indices are ingesting, the ingest load balances over available nodes anyway, each node with a few active indices. Since these tend to rollover at 50G in a day or two, searching is typically spread over the past week, so still multiple shards are involved in searches, and other users search other indices anyway, so search load is distributed.

[Phandora (Mayte)]

In our case, more than 95% of the documents are ingested in the same index, therefore setting only 1 primary shard will undercut the elasticsearch ingest performance.

Since the workload is variable (it always is), index management is a must to keep a good cluster performance. However, shrinking the index using ILM does not allow us to adjust the proper number of shards.

For instance:

Assuming 25gb as our desired shard size and a rollover policy with max_size=150gb and shrink with number_of_shards=3.

Index_1 --> primary_shards: 6; pri.store.size=150gb --> After shrink : too few shards
Index_2 --> primary_shards: 6; pri.store.size=70gb --> After shrink : desired shards
Index_3 --> primary_shards: 6; pri.store.size=20gb --> After shrink : too many shards