How to recover cluster state red in K8s

We are using Elasticsearch along with Graylog. We have three replicas running.
On the Graylog side I noticed that the Elasticsearch cluster status is red. Started to investigate, and noticed that we had run out of disk space. Allocated more space. And tried to restart the pods in Kubernetes, but the Elasticsearch wont start, as according the logs the cluster health is still red.

I can clarify this inside the pod, by checking the cluster health:

curl -X GET "localhost:9200/_cluster/health?pretty"
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 120,
"active_shards" : 120,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 98.36065573770492

How to recover the cluster state to green?

Welcome to our community! :smiley:

What's the output from the _cat/allocation?v and _cat/indices?v endpoints?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.