How to fix signal red?

Elastic Stack Elasticsearch
1

I am very new to Elastic.

The project, which I was assigned, uses Elastic and worked without problem, but from a few days ago we could not be able to reach Elastic.

When I command

curl http://localhost:9200/_cluster/health?pretty
{
"cluster_name" : "docker-cluster",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 121,
"active_shards" : 121,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 1615,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 1,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 6.954022988505747
}

curl -XGET localhost:9200/_cluster/allocation/explain?pretty
{
"index" : "ml_derivery_item_v2020_01_26__05_30_01",
"shard" : 4,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "CLUSTER_RECOVERED",
"at" : "2020-01-29T05:25:58.456Z",
"last_allocation_status" : "throttled"
},
"can_allocate" : "throttled",
"allocate_explanation" : "allocation temporarily throttled",
"node_allocation_decisions" : [
{
"node_id" : "hXHiSS7yQ7OYcyrikjvCNA",
"node_name" : "hXHiSS7",
"transport_address" : "172.21.0.2:9300",
"node_attributes" : {
"ml.machine_memory" : "8364023808",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"node_decision" : "throttled",
"store" : {
"in_sync" : true,
"allocation_id" : "NkILfO0vQrqARBIFFyA9aQ"
},
"deciders" : [
{
"decider" : "throttling",
"decision" : "THROTTLE",
"explanation" : "reached the limit of ongoing initial primary recoveries [4], cluster setting [cluster.routing.allocation.node_initial_primaries_recoveries=4]"
}
]
}
]
}

How can I fix this bug?

2

You don't mention if the 1-node cluster was recently restarted; I assume that this is the case. When a node restarts, it has to check all of the assigned shards (each index is made up of at least 1 primaray shard). Once the shard has been checked it becomes active. Depending on the amount of data in a shard, this can take some time.

Your cluster is still in the process of starting up shards. Up to 4 of these can be started in parallel, but more than 1600 are still waiting. Typically, a node should not have more than about 500 shards to function well. I'd recommend to add more data nodes to this cluster. With the default settings (1 primary, 1 replica shard per index), you'd have to account for a total number of about 3500 shards (on a 1 node cluster only primary shards can be activated, so replicas will suddenly become available, too). Without any further changes, you'd need 6-8 data nodes.

However, please check why there are so many shards / indices. If you're dealing with data on a time line (e.g. event logging), you should check if Elasticsearch's ILM feature could help keeping the number of indices low. For other use cases check if different indices with little data in each could be combined.

3

It looks like you have far too many shards for a cluster that size, so I would recommend reducing thst. This will also help sped up recovery. See this blog post for furtther details.

4

Thanks for adivice.
Yes. I did restart docker-cluster recently.

5

I am sorry. It is very basic though how can I reduce thst?

I have read that blog post, but I could not know how to do it.
Elastic is on docker with kibana in my case.

6

Once the cluster is healthy again you can change index templates to not create more primary shards than necessary (often 1) and reduce the number of new indices you are creating, e.g. by switching from daily to weekly or monthly indices. For existing indices you can either reindex into new indices with fewer shards or use the shrink index API.

7

I do not know how to reindex into new indices, so am going to try to use shrink index API following this link.
https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-shrink-index.html#shrink-index-api-prereqs

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.