Elastic Search Pagination in Term Aggrigation


(Ritesh Agarwal) #1

How can we achieve the Pagination in ES Term Aggregation to avoid Memory issues.


(Mark Harwood) #2

Check out the use of "partitioning" in terms aggregation or try the composite aggregation with the after param.


(Ritesh Agarwal) #3

The performance of both is slow. And when you do order by with them it's very slow.


(Mark Harwood) #4

Do you have numbers? Docs, ,nodes, response times


(Ritesh Agarwal) #5

"partitioning" does not work the way Pagination is required.

The Requirement is like One aggregation has 10K records. Then i want to make that as order by on some param then i want the 1K record in next request i could be able to get the next bunch of records.

To achieve this we need one parameter as "FROM", size is already there and i know we can set the limit but to get next bunch of records it is not possible.


(Mark Harwood) #6

In a distributed system compromises are required - physics gets in the way.
I don’t understand the details of your query/data - please share a simple example of JSON using just the core fields you care about. Just show the query that gets you the top N and we’ll figure out how to get the next N after that.


#7

Hi,

The pagination withe a From-Size approach is deprecated since Elasticsearch 5 if I remember well.

You should give a look to the scroll requests to slice the data.


(Mark Harwood) #8

Scroll is for retrieving docs, not aggregations (which are summaries derived from multiple docs).

Without seeing details of the query+data it's hard to determine if the required values are found in individual docs or combinations of multiple.


#9

Indeed you're right.
So it strongly depend of the how the data structure across indices and mapping, and the repartition across shards and nodes.
Sorry for the interference.


(Mark Harwood) #10

Sorry for the interference.

No worries - scroll may still be the right answer. "It depends" as we like to say around here.