Search scroll returning no hits (possible bug in 6.2?)

After an upgrade made to 6.2 our scrolling does no longer work. We are currently using a java client for it that follows the docs for first and subsequent requests (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html). To avoid any noise with the java client the requests are also done in the Kibana console and results confirm the issue. More details to follow

The example on the rest API is as follow:

First search request:

POST /cipher/_search?scroll=1m
{"size":500,"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"post_filter":{"has_child":{"query":{"bool":{"must":[{"bool":{"should":[{"term":{"processId":{"value":"511e1-v053-947a5-966d30281c4c16ad10c084533edd4a_511e1-v053-947a5-966d30281c4c16ad10c084533edd4a","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"match":{"clusterId":{"query":0,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":1,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":2,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":3,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":4,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":5,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":6,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":7,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":8,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":9,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"type":"process","score_mode":"none","min_children":0,"max_children":2147483647,"ignore_unmapped":false,"boost":1.0,"inner_hits":{"ignore_unmapped":false,"from":0,"size":3,"version":false,"explain":false,"track_scores":false}}},"explain":false,"_source":{"includes":["grantedTerritories","priorityDate","patentNumbers","accession","title","status"],"excludes":[]}}

It returns 500 results:

    {
      "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn",
      "took": 396,
      "timed_out": false,
      "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": 1075,
        "max_score": 1,
        "hits": [  {
    "_index": "cipher",
    "_type": "_doc",
    "_id": "5254018",
    "_score": 1,
    "_source": {
      "grantedTerritories": [
        "DE",
        "CH",
        "GB",
        "FR"
      ],
     ...]
      }
    }

Then it follows the search scroll:

POST  /_search/scroll 
{
    "scroll" : "1m", 
    "scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn"   
}

But it returns the following (please note the empty array of hits):

{
  "_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn",
  "took": 1,
  "timed_out": false,
  "terminated_early": true,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 1110,
    "max_score": 1,
    "hits": []
  }
}

What am I missing to get the next batch?

Also finding some issues on the ES host (self hosted, simply one node)

[2018-06-13T08:27:32,030][DEBUG][o.e.a.s.TransportSearchScrollAction] [masterdata-node-1] [14332] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException: [masterdata-node-1][192.168.3.187:9300][indices:data/read/search[phase/query/scroll]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [14332]
	at org.elasticsearch.search.SearchService.findContext(SearchService.java:520) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:391) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:410) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:407) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
[2018-06-13T08:27:32,030][DEBUG][o.e.a.s.TransportSearchScrollAction] [masterdata-node-1] [14333] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException: [masterdata-node-1][192.168.3.187:9300][indices:data/read/search[phase/query/scroll]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [14333]
	at org.elasticsearch.search.SearchService.findContext(SearchService.java:520) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:391) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:410) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.action.search.SearchTransportService$8.messageReceived(SearchTransportService.java:407) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:656) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.3.jar:6.2.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

Any help/hint would be really appreciated

This problem seems to be related with the "post_filter" object on the initial request, that in turn comes from doing a parent_child request, namely:

"post_filter":{"has_child":{"query":{"bool":{"must":[{"bool":{"should":[{"term":{"processId":{"value":"511e1-v053-947a5-966d30281c4c16ad10c084533edd4a_511e1-v053-947a5-966d30281c4c16ad10c084533edd4a","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"match":{"clusterId":{"query":0,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":1,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":2,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":3,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":4,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":5,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":6,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":7,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":8,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":9,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"type":"process","score_mode":"none","min_children":0,"max_children":2147483647,"ignore_unmapped":false,"boost":1.0,"inner_hits":{"ignore_unmapped":false,"from":0,"size":3,"version":false,"explain":false,"track_scores":false}}}

Still investigating/looking for a solution

It seems related with this issue: No hit using scan/scroll with has_parent filter

However I'm using 6.2 API, shouldn't that be fixed?
...
SearchRequest searchRequest = new SearchRequest(CIPHER_INDEX)
.source((new SearchSourceBuilder()
.query(QueryBuilders.boolQuery().must(getMatchQuery(searchFor)))
.postFilter(JoinQueryBuilders.hasChildQuery(ElasticConstants.CHILD_JOIN_REL,
filterByProcessAndCluster(reportMetadata.target().process(), processes, allClusters),
ScoreMode.None)
.innerHit(new InnerHitBuilder()))
.explain(false)))
.types(CIPHER_TYPE)
.searchType(SearchType.QUERY_THEN_FETCH);
...
elasticClient.search(searchRequest);
...
elasticClient.searchScroll(new SearchScrollRequest(result.getScrollId()))
...

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.