Elasticsearch crash unassigned all shards

Elastic Stack Elasticsearch
1

Hello.

I manage a cluster of 3 data nodes and 1 master.
Today the master have crashed with several exceptions:

  • a network issue ("fatal error on the network layer").
  • a StackOverflow: null

I was bulk inserting a few hundred documents max.

Upon restarting the master, I have seen that all the shards of the cluster have been unassigned!
There are 24k shards in total.
The unassigned.reason provided "CLUSTER_RECOVERED".
I don't see why this would happen.

After the restart there is a ClusterBlockException: "blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]".
And then an ElasticsearchTimeoutException "Timeout waiting for task".

So I leave the cluster to reassign the shards, but the operation is reaaally slow.
And it seem to get slower and slower with time.
The log shows a JvmGcMonitorService message about every minute that says ~300ms of collecting over the last second.

I tried to restart the master, hoping that whatever memory hog will go away.
But then the number of unassigned shards got back to 24k!
Is this normal? I would have thought that whatever assignation would have persisted.

Right now there is still 1k shards to assign, but I have no idea when it will finish.
There is less than one shard allocation per second.

And I am worried that on the next restart the issue will arise again.

I am running ES 5.6.3 on an Oracle JDK 8.
There is 31 GB of XMX (more than enough I think).
The CPU/memory of the machine is enough too.

Anyone has a clue on what is happening?

2

The first problem is that you have a single master eligible node. This is very bad and you should always aim to have at least three so that losing one node does not mean no master can be elected.

The second problem is that you have far, far too many shards in the cluster. This makes changes to the cluster state flow and is one of the reasons you are seeing very slow recovery. I would recommend you reduce the number of shards in the cluster by at least a factor of 20, ideally more.

3

Hello Christian.

Thank you for your assistance.
I will monitor the number of shards.

As for the number of masters, I am surprised they need to be that many. I would have tought that two is sufficient.
I never ran into this kind of stability issues until today.

By the way the shard assignment finished after more than 15 hours.
I have yet to restart the cluster to see if it happens again.

4

Elasticsearch leader election is based on consensus so a majority of master eligible nodes need to be present to elect a new one. You therefore need to get to at least three as the majority of 2 is 2.

Also have a look at this blog post which is very relevant for your issue.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.