# One of the cluster nodes get crashing down when cluster was allocating shards

**URL:** <https://discuss.elastic.co/t/one-of-the-cluster-nodes-get-crashing-down-when-cluster-was-allocating-shards/134350>\
**Category:** Elasticsearch\
**Created:** [June 4, 2018, 2:54am UTC](https://discuss.elastic.co/t/one-of-the-cluster-nodes-get-crashing-down-when-cluster-was-allocating-shards/134350 "2018-06-04T02:54:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vincentduan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vincentduan/32/30691_2.png) [@vincentduan](https://discuss.elastic.co/u/vincentduan)\
**Post date:** [June 4, 2018, 2:54am UTC](https://discuss.elastic.co/t/one-of-the-cluster-nodes-get-crashing-down-when-cluster-was-allocating-shards/134350/1 "2018-06-04T02:54:20Z")

</div>

Our cluster contains 3 master nodes and 4 data nodes ,with each node has 64G memory and 24 Cores.

We have 57 indexs and 570 shards (one primary and one replic), the number of docs is about 30 Billion.

The circumstances is following:

when I restart our cluster by some reason, the cluster is in yellow status and gets allocating.Everything seems to be fine. After several hours later , I found one of data nodes get crashed and the status of cluster is red.  
I used `jps` command in THE crashed node, it didn't show me the elasticsearch process.  
I am confused that I didn't do any operation and query to the cluster, only waiting it allocating, it still can get one nodes crashed.

THANK YOU FOR YOUR APPLY

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 5, 2018, 1:53pm UTC](https://discuss.elastic.co/t/one-of-the-cluster-nodes-get-crashing-down-when-cluster-was-allocating-shards/134350/2 "2018-06-05T13:53:57Z")

</div>

You probably have too many shards per node.

May I suggest you look at the following resources about sizing:

[https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing)

> **[How many shards should I have in my Elasticsearch cluster?
	  	 | Elastic](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)**
>
> Elasticsearch is a very versatile platform, that supports a variety of use cases, and provides great flexibility around data organisation and replication strategies. This flexibility can however somet...

> **[NetSecureDay: Managing your Black Friday Logs](https://speakerdeck.com/elastic/netsecureday-managing-your-black-friday-logs)**
>
> Surveiller une application complexe n’est pas une tâche aisée, mais avec les bons outils, ce n’est pas si sorcier. Néanmoins, des périodes fortes telles que les opérations de type « Black Friday » (Vendredi noir) ou période de Noël peuvent pousser...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 3, 2018, 1:54pm UTC](https://discuss.elastic.co/t/one-of-the-cluster-nodes-get-crashing-down-when-cluster-was-allocating-shards/134350/3 "2018-07-03T13:54:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.