After an upgrade made to 6.2 our scrolling does no longer work. We are currently using a java client for it that follows the docs for first and subsequent requests (https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html). To avoid any noise with the java client the requests are also done in the Kibana console and results confirm the issue. More details to follow
The example on the rest API is as follow:
First search request:
POST /cipher/_search?scroll=1m
{"size":500,"query":{"bool":{"must":[{"match_all":{"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"post_filter":{"has_child":{"query":{"bool":{"must":[{"bool":{"should":[{"term":{"processId":{"value":"511e1-v053-947a5-966d30281c4c16ad10c084533edd4a_511e1-v053-947a5-966d30281c4c16ad10c084533edd4a","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},{"bool":{"should":[{"match":{"clusterId":{"query":0,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":1,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":2,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":3,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":4,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":5,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":6,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":7,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":8,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}},{"match":{"clusterId":{"query":9,"operator":"OR","prefix_length":0,"max_expansions":50,"fuzzy_transpositions":true,"lenient":false,"zero_terms_query":"NONE","auto_generate_synonyms_phrase_query":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"type":"process","score_mode":"none","min_children":0,"max_children":2147483647,"ignore_unmapped":false,"boost":1.0,"inner_hits":{"ignore_unmapped":false,"from":0,"size":3,"version":false,"explain":false,"track_scores":false}}},"explain":false,"_source":{"includes":["grantedTerritories","priorityDate","patentNumbers","accession","title","status"],"excludes":[]}}
It returns 500 results:
{
"_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn",
"took": 396,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1075,
"max_score": 1,
"hits": [ {
"_index": "cipher",
"_type": "_doc",
"_id": "5254018",
"_score": 1,
"_source": {
"grantedTerritories": [
"DE",
"CH",
"GB",
"FR"
],
...]
}
}
Then it follows the search scroll:
POST /_search/scroll
{
"scroll" : "1m",
"scroll_id" : "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn"
}
But it returns the following (please note the empty array of hits):
{
"_scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAADX9Fkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA1_hZHNUhLT2hPelE5ZVBJY3FCaGd3MlZnAAAAAAAANf8WRzVIS09oT3pROWVQSWNxQmhndzJWZwAAAAAAADYAFkc1SEtPaE96UTllUEljcUJoZ3cyVmcAAAAAAAA2ARZHNUhLT2hPelE5ZVBJY3FCaGd3MlZn",
"took": 1,
"timed_out": false,
"terminated_early": true,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1110,
"max_score": 1,
"hits": []
}
}
What am I missing to get the next batch?