Elasticsearch cluster health is "status": "yellow"

Elastic Stack Elasticsearch
1

I have elasticsearch 6.x cluster and I just noticed that status is yellow

GET _cluster/health

{
  "cluster_name": "X",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 5,
  "active_primary_shards": 1911,
  "active_shards": 3824,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 1,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 99.97385620915033
}

How does one troubleshoot? How can I make it green again?

Please advise.

2

You probably need to wait until all shards are recovered.

BUT you probably have too many shards per node. You should reduce that number.

3

I did noticed that one of nodes was down and I restart it and even though I'm not seeing any "Shard Activity", cluster health still says yellow(

4

What are the pending tasks? https://www.elastic.co/guide/en/elasticsearch/reference/6.0/cat-pending-tasks.html

Can you run explain and paste the results here? https://www.elastic.co/guide/en/elasticsearch/reference/6.0/cluster-allocation-explain.html#_explain_api_request

5

@dadoonet

cat pending tasks | Elasticsearch Reference [6.0] | Elastic

# curl --silent --request GET --header 'Content-Type: application/json' $ELASTICSEARCH_URI/_cat/pending_tasks?v
insertOrder timeInQueue priority source
#

Cluster Allocation Explain API | Elasticsearch Reference [6.0] | Elastic

# curl --silent --request GET --header 'Content-Type: application/json' $ELASTICSEARCH_URI/_cluster/health/explain?pretty
{
  "cluster_name" : "X",
  "status" : "red",
  "timed_out" : true,
  "number_of_nodes" : 5,
  "number_of_data_nodes" : 5,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
#

this time it said "red", but I'm looking in Kibana and it says yellow

6

possibly related to: Kibana Monitoring App reports Elasticsearch as Version: 6.0.0 instead of 6.0.1 - X-Pack - Discuss the Elastic Stack

as I restarted one of the nodes that was 6.0.0, now all nodes are on 6.0.1 and there are no unassigned_shards anymore, and status back to GREEN!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.