Slow index creation and indexing when many indices must be created at once

It actually does, but perhaps not in the way you might think.

Each shard has a management cost in Elasticsearch. At a certain point, a significant portion of your heap will be locked up, just for keeping tabs on your shards. When that happens, everything slows down. Indexing takes a hit. Any operation that requires a read from or write to the cluster metadata will be slowed down, including adding more indices.

There are no hard rules here, but with a 30G heap, I would not want more than 900 - 1200 shards per node. 600 is a pretty safe bet. That safe number falls off pretty dramatically if your heap sizes are smaller than that.

You can complain that 12 shards initializing at once is limiting you, but the recommendation to reduce shard count will save you a lot of headache down the road.

1 Like