# Cluster Blackout

**URL:** <https://discuss.elastic.co/t/cluster-blackout/116886>\
**Category:** Elasticsearch\
**Created:** [January 24, 2018, 3:42pm UTC](https://discuss.elastic.co/t/cluster-blackout/116886 "2018-01-24T15:42:16Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Junaid03](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@Junaid03](https://discuss.elastic.co/u/Junaid03)\
**Post date:** [January 24, 2018, 3:42pm UTC](https://discuss.elastic.co/t/cluster-blackout/116886/1 "2018-01-24T15:42:17Z")

</div>

Recently one of my nodes which was running on a 2g memory set for both Xms and Xmx, went crazy doing gc:

```
[2018-01-21T21:29:05,958][WARN][o.e.m.j.JvmGcMonitorService] [NVMBD2BFM70V03] [gc][young][4109765][392598] duration [13.5s], collections [1]/[3.9s], total [13.5s]/[50.5m], memory [1.4gb]->[1.5gb]/[1.9gb], all_pools {[young] [456.8mb]->[543.3mb]/[546.1mb]}{[survivor] [9mb]->[9mb]/[68.2mb]}{[old] [1gb]->[1gb]/[1.3gb]}
[2018-01-21T21:29:07,500][WARN][o.e.m.j.JvmGcMonitorService] [NVMBD2BFM70V03] [gc][4109765] overhead, spent [13.5s] collecting in the last [3.9s]

```

The setup I have consists of 9 nodes:

```
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
x 4 97 4 0.85 0.45 0.36 di - x
x 2 66 3 0.54 0.31 0.30 mi - x
x 14 99 9 0.52 0.69 0.88 d - x
x 13 99 4 0.71 0.41 0.35 di - x
x 2 67 3 0.78 0.69 0.55 mi * x
x 8 74 11 0.25 0.86 1.48 d - x
x 1 66 1 0.05 0.07 0.06 - - x
x 5 73 9 1.76 1.23 1.05 di - x
x 2 63 2 0.33 0.34 0.31 mi - x

```

Now one node as it went down should not have bought the entire cluster down, however when I dig deep into logs I see the malfunctioning node was rightly evicted, the master node at the time had the following logs:

```
$ cat MACHINELEARNING-2018-01-21.log

```

[HERE IS THE LOG (NOT ABLE TO FIT IN THE BODY HERE)](https://gist.github.com/Junaidsakib/6547c777ce1036f11b1e558aa4ec020a)

One other master eligible node also has a similar message:

```
$ cat MACHINELEARNING-2018-01-21.log
[2018-01-21T21:34:01,076][INFO][o.e.c.s.ClusterService] [NVMBD2BFL90V01] removed {{NVMBD2BFM70V03}{RKBjq_CxTFOAteqtLgHZwQ}{laMJDPWjQDWO5dKH07i-uw}{10.141.172.110}{10.141.172.110:9300},}, reason: zen-disco-receive(from master [master {NVMBD2BFL90V02}{oljvKtCNSbOHdNC_gJI12g}{VB7Ctc4FSgihC9hY1wrWNA}{10.141.172.106}{10.141.172.106:9300} committed version [259]])

```

The entire cluster at this moment went into hung state and remained so for the next six hours until I did a manual restart of the problematic "Crazy GC node".

What settings am I missing here?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 21, 2018, 3:42pm UTC](https://discuss.elastic.co/t/cluster-blackout/116886/2 "2018-02-21T15:42:37Z")

</div>

Closed on February 21, 2018, 3:42pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.