Max number of indices

The primary concern is more for shard count per node, rather than total number of indices. An acceptable number of shards per node can vary greatly depending on many factors.

Generally speaking, with data nodes having 30g heaps, you should be safe with 600 to 1,000 shards per node. While the cluster can operate safely with more than that in many instances, you would need to pay very close attention to the heap availability in Monitoring. If you are able to still maintain a gentle sawtooth pattern with the peaks at around 70% of the heap, and the troughs around 50%, the node is healthy (with regards to memory usage). If, however, the peaks in the graph are consistently over 70%, and staying there, rather than creating that sawtooth pattern, or the garbage collection is occurring too frequently, and not freeing up enough memory, then that will be an indication that your cluster + nodes are experiencing memory pressure.

This may be familiar to you already, and if so, you're well versed. The thing to be aware of is that having an increased number of shards per node will increase memory pressure. Keeping your cluster healthy with more indices (shards per node) frequently becomes a heap management exercise.

1 Like