Hi everyone.
I am having some questions about pagination, increasing the number or results etc.
I have couple of milions of items (products) in the engine. I would like to be able to paginate to all the results. Right now the limitations are to return max 100k of documents (1000 per page, 100 pages), if you want to return the rest of it, i have to use multi search, append some filter to each of the query, to achive each query response <1000 items.
The problem is, i do not know to what field i should filter. i can create "dummy" field like "bucket", and assign first 10k products to each of the bucket, but later, when i want to go to the page eg 130, i do not know which buckets should i query. If i have the sort based on some field, i need to query all, sort it out and return just the partial of this.
In my case, i get up to 100 multisearches for one results page. ( plus some for disjunctive faceting - facedfilters searches)
How can speed up the pagination, without having a lot of multisearches? How its done this on websites, where you need to show all the results (for crawlers,..)
Thank you for help.