# Elastic Search sudden blackout / cluster down

**URL:** <https://discuss.elastic.co/t/elastic-search-sudden-blackout-cluster-down/116467>\
**Category:** Elasticsearch\
**Created:** [January 22, 2018, 10:21am UTC](https://discuss.elastic.co/t/elastic-search-sudden-blackout-cluster-down/116467 "2018-01-22T10:21:25Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Junaid03](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@Junaid03](https://discuss.elastic.co/u/Junaid03)\
**Post date:** [January 22, 2018, 10:21am UTC](https://discuss.elastic.co/t/elastic-search-sudden-blackout-cluster-down/116467/1 "2018-01-22T10:21:25Z")

</div>

Hello everyone! Greetings!

I am running an elastic cluster with 3 masters and 5 data nodes, I have one coordinator node running on localhost which backsup as kibana fetch node (I mean it supports searches across the cluster)

My use case here is that I use elastic as a logging mechanism, whatever queries my server receives I bundle them up into the elastic cluster.

However one mistake I did was that I defined only one data node as es.hosts = [xxx.xxx.xxx](http://xxx.xxx.xxx)

Worthy to mention I am using spark to put everything into elastic:

JavaEsSpark.saveJsonToEs(  
dataset.toJavaRDD(),  
getESIndex(API\_TYPE.PREDICT),  
ImmutableMap.of("[es.mapping.id](http://es.mapping.id)", \_ESID));

Now yesterday on the weekend lots of load increased on my system and the system was in a hung state for over 12 hours, once I came to realize this I checked the logs and found:

[WARN][o.e.m.j.JvmGcMonitorService] [NVMBD2BFM70V03] [gc][4109061] overhead, spent [956ms] collecting in the last [1.6s]

Apart from increasing the Java memory on this node how do I prevent any such occurrences. What principles are to be followed here? Also I think elastic should have detected my whole cluster and not relied on one single node of failure, what is the config for that?

Also below listed is my cluster health:

{"cluster\_name":"MACHINELEARNING","status":"green","timed\_out":false,"number\_of\_nodes":9,"number\_of\_data\_nodes":5,"active\_primary\_shards":25,"active\_shards":50,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":0,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":100.0}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 19, 2018, 10:21am UTC](https://discuss.elastic.co/t/elastic-search-sudden-blackout-cluster-down/116467/2 "2018-02-19T10:21:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.