We are using Elasticsearch along with Graylog. We have three replicas running.
On the Graylog side I noticed that the Elasticsearch cluster status is red. Started to investigate, and noticed that we had run out of disk space. Allocated more space. And tried to restart the pods in Kubernetes, but the Elasticsearch wont start, as according the logs the cluster health is still red.
I can clarify this inside the pod, by checking the cluster health:
curl -X GET "localhost:9200/_cluster/health?pretty"
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 120,
"active_shards" : 120,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 98.36065573770492
}
How to recover the cluster state to green?
