# Elasticsearch master node was suddenly down

**URL:** <https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252>\
**Category:** Elasticsearch\
**Created:** [September 12, 2018, 3:22am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252 "2018-09-12T03:22:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wyukawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wyukawa/32/13336_2.png) [@wyukawa](https://discuss.elastic.co/u/wyukawa)\
**Post date:** [September 12, 2018, 3:22am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252/1 "2018-09-12T03:22:08Z")

</div>

move from [https://github.com/elastic/elasticsearch/issues/33618](https://github.com/elastic/elasticsearch/issues/33618)

We have three master nodes and one of them was suddenly down but there is no meaningful log.  
I guess `ExitOnOutOfMemoryError` worked.

Elasticsearch version is 6.4.0.

I analyze heap dump with MAT.

Here is the result.

```auto
  at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16([BII[C)I (UnicodeUtil.java:612)
  at org.apache.lucene.util.BytesRef.utf8ToString()Ljava/lang/String; (BytesRef.java:140)
  at org.elasticsearch.search.DocValueFormat$1.format(Lorg/apache/lucene/util/BytesRef;)Ljava/lang/String; (DocValueFormat.java:100)
  at org.elasticsearch.search.DocValueFormat$1.format(Lorg/apache/lucene/util/BytesRef;)Ljava/lang/Object; (DocValueFormat.java:77)
  at org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKeyAsString()Ljava/lang/String; (StringTerms.java:83)
  at org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKey()Ljava/lang/Object; (StringTerms.java:64)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalTerms.java:274)
  at org.elasticsearch.search.aggregations.InternalAggregation.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalAggregation.java:135)
  at org.elasticsearch.search.aggregations.InternalAggregations.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (InternalAggregations.java:77)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/bucket/terms/InternalTerms$Bucket; (InternalTerms.java:142)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalTerms.java:286)
  at org.elasticsearch.search.aggregations.InternalAggregation.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalAggregation.java:135)
  at org.elasticsearch.search.aggregations.InternalAggregations.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (InternalAggregations.java:77)
  at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(Ljava/util/List;Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (SearchPhaseController.java:525)
  at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(Ljava/util/Collection;Ljava/util/List;Ljava/util/List;Lorg/elasticsearch/action/search/SearchPhaseController$TopDocsStats;IZ)Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:502)
  at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(Ljava/util/Collection;ZZ)Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:419)
  at org.elasticsearch.action.search.SearchPhaseController$1.reduce()Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:738)
  at org.elasticsearch.action.search.FetchSearchPhase.innerRun()V (FetchSearchPhase.java:101)
  at org.elasticsearch.action.search.FetchSearchPhase.access$000(Lorg/elasticsearch/action/search/FetchSearchPhase;)V (FetchSearchPhase.java:44)
  at org.elasticsearch.action.search.FetchSearchPhase$1.doRun()V (FetchSearchPhase.java:86)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun()V (ThreadContext.java:723)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37)
  at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun()V (TimedRunnable.java:41)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624)
  at java.lang.Thread.run()V (Thread.java:748)

```

Any suggestions?

---

<div class="post-metadata">

**Author:** ![Dilip\_Kumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dilip_kumar/32/15459_2.png) [@Dilip\_Kumar](https://discuss.elastic.co/u/Dilip_Kumar)\
**Post date:** [September 12, 2018, 3:39am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252/2 "2018-09-12T03:39:01Z")

</div>

`ExitOnOutOfMemoryError` means some activity or query taking too much memory from available memory.. check your mlock and check you heap memory for this, if more memory needed than need to increase. you can check by some command  
curl x.x.1.x:9200/\_cat/nodes?h=heap\*&v  
curl x.x.1.x:9200/\_cat/nodes?v

---

<div class="post-metadata">

**Author:** ![wyukawa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wyukawa/32/13336_2.png) [@wyukawa](https://discuss.elastic.co/u/wyukawa)\
**Post date:** [September 12, 2018, 3:47am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252/3 "2018-09-12T03:47:42Z")

</div>

Thank you for reply.  
The java heap memory didn't change so much at that time.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 10, 2018, 3:47am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-was-suddenly-down/148252/4 "2018-10-10T03:47:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.