Help with cluster configuration for a deployment moving from 2.4 in AWS to 7.7 in elastic cloud

[Michael_Turbe (Michael Turbé)]

Hi all, we currently have a large elasticsearch 2.4 deployment that's been running successfully, hosted in AWS, for the last 3 years. The cluster is comprised of ~15,000 indexes, 7700 active primary shards, and 15400 active shards. The cluster has 3 nodes with 300gb of disk on each. Each index has a completely different schema with many columns (up to 4000); we're in the business of customer driven form building. As far as I recall, I never did any kind of cluster configuration beyond the default.

We're trying to migrate to elasticssearch 7.7 in elastic cloud. All of the query/schema building configuration is taken care of and working. Right off the bat, while attempting to do a fresh build of the data, we run up against: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [2000]/[2000] maximum shards open

After doing some reading on cluster configuration, it's clear that our dataset with so many wide indexes is not going to be easy to set up. I've looked at variables like shard size, heap size, and maximum shards per node, but so far have not landed on a configuration that will successfully index all the data.

Given what I've stated about our data, does anyone have thoughts on a good cluster configuration that would work?

I appreciate the help.

[Christian_Dahlqvist]

Having very large number of shards an indices with very large number of fields are both things that has been known to cause a lot of problems, which is why protections have been added to Elasticsearch over time.

If you were hosting the cluster yourself you would be able to override these, but I am not sure to what level that is allowed on a hosted service. I do not know the use case so can not suggest alternate approaches, and you will need to check with service provider wherever relaxing these constraints is an option or not.