Index Lifecycle Management - closing indices

The question is why you need to close indices at all anymore.

This was a frequently used feature in order to keep the total open shard count below the "20 per gigabyte of heap" max that was preached as gospel since Elasticsearch 1.7/2.x. However, many of the reasons this was necessary have been reduced or outright fixed/removed since then.

In fact, in 8.3, the "20 shards per gigabyte of heap" guidance was officially replaced. The new guidance is here, but the paragraph addressing this particular bit is "Data nodes should have at least 1kB of heap per field per index, plus overheads":

The exact resource usage of each mapped field depends on its type, but a rule of thumb is to allow for approximately 1kB of heap overhead per mapped field per index held by each data node. You must also allow enough heap for Elasticsearch’s baseline usage as well as your workload such as indexing, searches and aggregations. 0.5GB of extra heap will suffice for many reasonable workloads, and you may need even less if your workload is very light while heavy workloads may require more.

For example, if a data node holds shards from 1000 indices, each containing 4000 mapped fields, then you should allow approximately 1000 × 4000 × 1kB = 4GB of heap for the fields and another 0.5GB of heap for its workload and other overheads, and therefore this node will need a heap size of at least 4.5GB.

Note that this rule defines the absolute maximum number of indices that a data node can manage, but does not guarantee the performance of searches or indexing involving this many indices. You must also ensure that your data nodes have adequate resources for your workload and that your overall sharding strategy meets all your performance requirements.

You can confirm the change on this page by pulling up the older documentation in the drop-down.

1 Like