I try to implement pagination with using scroll. I read this article but does not help me.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-scroll.html
My problem is when I try to make a few requests the results are same. Scrolling is not working.
We have more than 20 billions records in the database.
My ElasticSearch version is: 6.6.1
PHP code snippet:
public function someFunction($client, $elasticResult, $limit)
{
$requestQuery = {
"scroll":"1m",
"size":10000,
"index":"my_index",
"type":"my_type",
"body":{
"query":{
"bool":{
"must":{
"terms":{
"db":[ 13, 15, 19
]
}
}
}
},
"sort":[
"_doc"
]
}
}
$client = $client->build();
$response = $client->search($requestQuery);
$elasticResult->setScrollTime('1m');
$metadata = [];
yield from $elasticResult->getPagingResults($response, $client, $metadata, $limit);
}
public function getPagingResults(array $response, Client $client, &$result, int $limit = null): \Generator
{
$i = 0;
while (isset($response['hits']['hits']) && \count($response['hits']['hits']) > 0) {
foreach ($response['hits']['hits'] as $hit) {
++$i;
if (null !== $limit && $i > $limit) {
break;
}
if (!empty($hit['_source'])) {
foreach ($hit['_source'] as $key => $value) {
if (!isset($result[$hit['_source']['db']][$key])) {
$result[$hit['_source']['db']][$key] = 0;
}
if ('' !== trim($value)) {
++$result[$hit['_source']['db']][$key];
}
}
}
yield $hit;
}
if (null !== $limit && $i > $limit) {
break;
}
$scrollId = $response['_scroll_id'];
$response = $client->scroll([
'scroll_id' => $scrollId,
'scroll' => '1m',
]);
}
}
If is necessary additional info, Please don't hesitate to ask.