One of the cluster nodes get crashing down when cluster was allocating shards

Elastic Stack Elasticsearch
1

Our cluster contains 3 master nodes and 4 data nodes ,with each node has 64G memory and 24 Cores.

We have 57 indexs and 570 shards (one primary and one replic), the number of docs is about 30 Billion.

The circumstances is following:

when I restart our cluster by some reason, the cluster is in yellow status and gets allocating.Everything seems to be fine. After several hours later , I found one of data nodes get crashed and the status of cluster is red.
I used jps command in THE crashed node, it didn't show me the elasticsearch process.
I am confused that I didn't do any operation and query to the cluster, only waiting it allocating, it still can get one nodes crashed.

THANK YOU FOR YOUR APPLY

2

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.