Elasticsearch master node was suddenly down

Elastic Stack Elasticsearch
1

move from https://github.com/elastic/elasticsearch/issues/33618

We have three master nodes and one of them was suddenly down but there is no meaningful log.
I guess ExitOnOutOfMemoryError worked.

Elasticsearch version is 6.4.0.

I analyze heap dump with MAT.

Here is the result.

  at org.apache.lucene.util.UnicodeUtil.UTF8toUTF16([BII[C)I (UnicodeUtil.java:612)
  at org.apache.lucene.util.BytesRef.utf8ToString()Ljava/lang/String; (BytesRef.java:140)
  at org.elasticsearch.search.DocValueFormat$1.format(Lorg/apache/lucene/util/BytesRef;)Ljava/lang/String; (DocValueFormat.java:100)
  at org.elasticsearch.search.DocValueFormat$1.format(Lorg/apache/lucene/util/BytesRef;)Ljava/lang/Object; (DocValueFormat.java:77)
  at org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKeyAsString()Ljava/lang/String; (StringTerms.java:83)
  at org.elasticsearch.search.aggregations.bucket.terms.StringTerms$Bucket.getKey()Ljava/lang/Object; (StringTerms.java:64)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalTerms.java:274)
  at org.elasticsearch.search.aggregations.InternalAggregation.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalAggregation.java:135)
  at org.elasticsearch.search.aggregations.InternalAggregations.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (InternalAggregations.java:77)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms$Bucket.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/bucket/terms/InternalTerms$Bucket; (InternalTerms.java:142)
  at org.elasticsearch.search.aggregations.bucket.terms.InternalTerms.doReduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalTerms.java:286)
  at org.elasticsearch.search.aggregations.InternalAggregation.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregation; (InternalAggregation.java:135)
  at org.elasticsearch.search.aggregations.InternalAggregations.reduce(Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (InternalAggregations.java:77)
  at org.elasticsearch.action.search.SearchPhaseController.reduceAggs(Ljava/util/List;Ljava/util/List;Lorg/elasticsearch/search/aggregations/InternalAggregation$ReduceContext;)Lorg/elasticsearch/search/aggregations/InternalAggregations; (SearchPhaseController.java:525)
  at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(Ljava/util/Collection;Ljava/util/List;Ljava/util/List;Lorg/elasticsearch/action/search/SearchPhaseController$TopDocsStats;IZ)Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:502)
  at org.elasticsearch.action.search.SearchPhaseController.reducedQueryPhase(Ljava/util/Collection;ZZ)Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:419)
  at org.elasticsearch.action.search.SearchPhaseController$1.reduce()Lorg/elasticsearch/action/search/SearchPhaseController$ReducedQueryPhase; (SearchPhaseController.java:738)
  at org.elasticsearch.action.search.FetchSearchPhase.innerRun()V (FetchSearchPhase.java:101)
  at org.elasticsearch.action.search.FetchSearchPhase.access$000(Lorg/elasticsearch/action/search/FetchSearchPhase;)V (FetchSearchPhase.java:44)
  at org.elasticsearch.action.search.FetchSearchPhase$1.doRun()V (FetchSearchPhase.java:86)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun()V (ThreadContext.java:723)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37)
  at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun()V (TimedRunnable.java:41)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1149)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:624)
  at java.lang.Thread.run()V (Thread.java:748)

Any suggestions?

2

ExitOnOutOfMemoryError means some activity or query taking too much memory from available memory.. check your mlock and check you heap memory for this, if more memory needed than need to increase. you can check by some command
curl x.x.1.x:9200/_cat/nodes?h=heap*&v
curl x.x.1.x:9200/_cat/nodes?v

3

Thank you for reply.
The java heap memory didn't change so much at that time.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.