How to recover a failed data node in Production cluster?

Elastic Stack Elasticsearch
1

What steps must to be performed to recover a failed node in a cluster?

2

You should just be able to restart it and it'll rejoin.

Though that may depend on the failure mode.

3

What if

unassigned_shards: > 0
OR
initializing_shards: > 0

OR in general, Cluster health is RED or Yellow

4

If it's initialising then just wait for everything to finish.
If they're unassigned then, it depends.

What version are you on?

5

We are still on old version 1.7.5 though we are planning to upgrade.

This is our current status:

cluster_name: "abc",
status: "red",
timed_out: false,
number_of_nodes: 2,
number_of_data_nodes: 2,
active_primary_shards: 4226,
active_shards: 8452,
relocating_shards: 0,
initializing_shards: 0,
unassigned_shards: 3781,
delayed_unassigned_shards: 0,
number_of_pending_tasks: 0,
number_of_in_flight_fetch: 0

6

That's faaaaaaar too many shards and likely to be causing your issues.

7

We have got around 800+ indices. How can this be resolved ?

8

Reindex. You will need to do that to get to 5.X anyway, so it might make sense to start a 5.X cluster, then run a remote index to pull the data from that old cluster - assuming you still want it.

9

Thanks @warkolm. I will get back if I have any further questions.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.