Elasticsearch cluster is Red , the service is still running

[linuxhustler (Nishyant Jani)]

Elasticsearch contains the following: 1 cluster, 1 node, 1 shard, 0 replicas. Added ASA's to send logs to the server and ran out of disk space. Added disk space to the cloud server & tried to reroute it to the primary shard but it did not work. It shows that there are unassigned shards, but it is not assigning the when instructed using the following commands.

curl -XPOST ‘192.168.10.10:9200/_cluster/reroute?pretty’ -H ‘Content-Type: application/json’ -d’
{
“commands” : [
{
“allocate_stale_primary” : {
“index” : “graylog_0”, “shard” : 0,
“node” : “graylog-node1”,
“accept_data_loss” : true
}
}
]
}
’

Output displayed on the CLI:
“acknowledged” : true,
“state” : {
“version” : 4,
“state_uuid” : “”,
“master_node” : “”,
“blocks” : { },
“nodes” : {
"" : {
“name” : “graylog-node1”,
“ephemeral_id” : “”,
“transport_address” : “192.168.10.10:9300”,
“attributes” : { }
}
},
“routing_table” : {
“indices” : {
“graylog_0” : {
“shards” : {
“0” : [
{
“state” : “INITIALIZING”,
“primary” : true,
“node” : “”,
“relocating_node” : null,
“shard” : 0,
“index” : “graylog_0”,
“recovery_source” : {
“type” : “EXISTING_STORE”
},
“allocation_id” : {
“id” : “”
},
“unassigned_info” : {
“reason” : “CLUSTER_RECOVERED”,
“at” : “2017-10-31T14:16:23.605Z”,
“delayed” : false,
“allocation_status” : “no_valid_shard_copy”
}
}
]
}
}
}
},
“routing_nodes” : {
“unassigned” : [ ],
“nodes” : {
"" : [
{
“state” : “INITIALIZING”,
“primary” : true,
“node” : “”,
“relocating_node” : null,
“shard” : 0,
“index” : “graylog_0”,
“recovery_source” : {
“type” : “EXISTING_STORE”
},
“allocation_id” : {
“id” : “”
},
“unassigned_info” : {
“reason” : “CLUSTER_RECOVERED”,
“at” : “2017-10-31T14:16:23.605Z”,
“delayed” : false,
“allocation_status” : “no_valid_shard_copy”
}
}
]
}
}
}
}

Also,

curl -XGET ‘http://192.168.10.10:9200/_cluster/health?pretty=true’ ====> as shown below

{
“cluster_name” : “graylog”,
“status” : “red”,
“timed_out” : false,
“number_of_nodes” : 1,
“number_of_data_nodes” : 1,
“active_primary_shards” : 0,
“active_shards” : 0,
“relocating_shards” : 0,
“initializing_shards” : 0,
“unassigned_shards” : 1,
“delayed_unassigned_shards” : 0,
“number_of_pending_tasks” : 0,
“number_of_in_flight_fetch” : 0,
“task_max_waiting_in_queue_millis” : 0,
“active_shards_percent_as_number” : 0.0
}

Also,

curl -s -XGET http://192.168.10.10:9200/_cat/shards ====> as shown below

graylog_0 0 p UNASSIGNED

Need help to troubleshoot and learn from this ?