ES crash with fatal error on search thread

Elastic Stack Elasticsearch
1

hi,
I tried a query with AND operator but ES crashed everytime with the following stacktrace error:

[2020-10-06T14:45:54,221][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] fatal error in thread [elasticsearch[node1][search][T#6]], exiting
java.lang.StackOverflowError: null
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:477) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) ~[?:1.8.0_144]
	at org.elasticsearch.index.search.NestedHelper.mightMatchNestedDocs(NestedHelper.java:77) ~[elasticsearch-6.8.1.jar:6.8.1]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) ~[?:1.8.0_144]
	at org.elasticsearch.index.search.NestedHelper.mightMatchNestedDocs(NestedHelper.java:87) ~[elasticsearch-6.8.1.jar:6.8.1]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) ~[?:1.8.0_144]
	at org.elasticsearch.index.search.NestedHelper.mightMatchNestedDocs(NestedHelper.java:82) ~[elasticsearch-6.8.1.jar:6.8.1]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) ~[?:1.8.0_144]
	at org.elasticsearch.index.search.NestedHelper.mightMatchNestedDocs(NestedHelper.java:87) ~[elasticsearch-6.8.1.jar:6.8.1]
	at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[?:1.8.0_144]
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) ~[?:1.8.0_144]
	at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) ~[?:1.8.0_144]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:1.8.0_144]
	at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454) ~[?:1.8.0_144]
	at org.elasticsearch.index.search.NestedHelper.mightMatchNestedDocs(NestedHelper.java:82) ~[elasticsearch-6.8.1.jar:6.8.1]

is there anyone that can explain to me it? only DATA nodes go down.

2

Can you share the query, a sample index with mapping and a sample document? Can you reproduce this on a new cluster?

Also, is it possible to upgrade to the latest 6.8 version on your side and test this as well, if the problem still exists?

3

this is a sample query:

GET index-provider_data_all/_count
{"query": {"bool": {"filter": {"term": {"__countryCode": "XXX"}},"must": [{"simple_query_string" : {"query": "Eyüp Doe|DoeTest Bar|RamDoe Sharma","fields": ["name"],"default_operator": "AND"}}]}}}

nested query is composed by 1023 "OR" operators.

this is a sample index with mapping:

{
  "index": {
    "mappings": {
      "data": {
        "properties": {
          "__countryCode": {
            "type": "keyword"
          },
          "__providerCode": {
            "type": "keyword"
          },
          "name": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
      }
    }
  }
}

I reproduced the same error on two different clusters with 2 different versions v6.8.1 and v6.8.6.

currently, I cannot upgrade it to 6.8.12 :frowning:

4

I am confused, the query is an AND query, but you mention a nested query with 1023 or operators. Would it be possible to create a full reproduction including index creation with mapping, document indexing and query so we can follow along?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.