One Huge Index or Multiple Indexes?

Elastic Stack Elasticsearch
1

Our company has a very large Elasticsearch index in production. The cluster holds HTML documents that are large. The index has 600 shards in it. The largest shard has 1000GB in it. We are planning a rework of our system in a entirely new cluster. We are planning on using HTML minimization to make each HTML document smaller (about 50% - 70%). We are also wanting to add more shards to the cluster. As Elasticsearch has a default max shard total of 1000 per index, we have two options:

  • Bump up the max shard total from 1000 to 4000
  • Create 4 indexes of 1000 shards each. We will query the index using an alias.

Which path do you think is the better one to take?

2

From Elastic Search to Elasticsearch

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.