ES has a lot of unassigned_shards, failed to process cluster event, stuck in 503 error

Elastic Stack Elasticsearch
1

curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "es",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 23,
"number_of_data_nodes" : 20,
"active_primary_shards" : 3491,
"active_shards" : 5833,
"relocating_shards" : 0,
"initializing_shards" : 46,
"unassigned_shards" : 1345
}
ES has a lot of unassigned_shards, failed to process cluster event, stuck in 503 error
How to solve this? Thanks

2

First of all, we need some more information.

  • What version of Elasticsearch are you running?
  • Are you really sure, all nodes are running the same version?
  • What Java version are you running?
  • Are you really sure all nodes are running the same version?
  • How many Nodes and Indices and Shards should be there? We only see how many are there now
  • What do the logs of the nodes say? Of the connected ones and the possibly unconnected ones.
  • How did this happen? Out of blue sky? During an upgrade? During a reboot?
3

--All ES "version" : {
"number" : "1.3.5",
"build_hash" : "4a50e7df768fddd572f48830ae9c35e4ded86ac1",
"build_timestamp" : "2014-11-05T15:21:28Z",
"build_snapshot" : false,
"lucene_version" : "4.9"
},
--All java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)
--No upgrade or reboot or any other operation.

What could be the reason? Thanks!

4

Answering the other questions will provide us context, simply posting what you have so far tells us nothing.