Shards stuck in relocating

Elastic Stack Elasticsearch
1

We had a cluster go into recovery today due to an outage. It has recovered, but there seems to be 2 shards stuck in relocation. The cluster is green, but search is extremely slow to non-responsive (Kibana times out). We have shut down indexing in the meantime.

-XGET 'http://localhost:9200/_cluster/health?pretty'

returns

{
"cluster_name" : "elasticsearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 5,
"active_primary_shards" : 6161,
"active_shards" : 12322,
"relocating_shards" : 2,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1684060,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 38878795,
"active_shards_percent_as_number" : 100.0
}

We are running ES 2.0

Where should we start troubleshooting?

2

Though it may or may not be directly related to the issues you are having, I would recommend you reconsider your indexing strategy, because that is a very large number of shards for a cluster that size.

3

Yeah thanks, this is a proof of concept currently, so I've been using ES defaults (5shards/index), which has quickly blown up when combined with rolling daily indexes. Definitely need to address that issue.

In regards to this issue, I bounced the master node, and every re-distributed and ended up fixing itself.