Total shards, but this cluster currently has [1000]/[1000]

[rbrodrigues (Rogerio Rodrigues)]

Hello Elasticsearch Team,

I have the following problem when I index a very large volume of data.

2019-10-23 17:33:52,158 ERROR 0:0:0:0:0:0:0:1 admin    [LumisRequest - http-nio-8080-exec-7; url: [http://localhost:8080/portal/lumis/service/portalmanagement/servicesearchindex/DouiReindexerControllerXml.jsp]; login: [admin]] lumis.portal.bigdata.elasticsearch.ElasticsearchRepository - Could not update mapping of document type lumis_service_forum_message. Its indices ([lumisportal-null-lumis_service_forum_message, lumisportal-es-lumis_service_forum_message, lumisportal-pt_br-lumis_service_forum_message, lumisportal-en_us-lumis_service_forum_message]) will be recreated, because 'force' is 'true', and will require reindexing.
java.lang.IllegalArgumentException: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validateIndexSettings(MetaDataCreateIndexService.java:629)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService.validate(MetaDataCreateIndexService.java:618)
	at org.elasticsearch.cluster.metadata.MetaDataCreateIndexService$IndexCreationTask.execute(MetaDataCreateIndexService.java:286)
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:687)
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:310)
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:210)
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:142)
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Specifically in this passage

java.lang.IllegalArgumentException: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;

How to solve this problem?

I have two nodes.

[Christian_Dahlqvist]

There is a soft limit on the number of shards per data node which is set rather high at 1000. You say you have 2 nodes. How many of these are data nodes?

Having large numbers of small indices and shards can cause performance problems and instability so I would recommend you look into reducing the number of shards in your cluster. I would also recommend you read this blog post.