How to get a page from elasticsearch where from parameter is greater than 10000

[Adarsh_Bajpai]

Hi All,

In our application, we have a listing of elasticsearch data, the data is rendered with pagination. Users can click on any page to get listing of that particular page.

After upgrade to 5.6, we are not able to get the result if the user clicks on a page greater than 10000.

As elasticsearch has mentioned the use of scroll API for deep pagination to get more record than 10000, but in my case scroll will not work as the user can randomly click on any page.

Can anyone suggest me how can I achieve deep pagination for users in the current version of elasticsearch. Thanks in advance.

[dadoonet (David Pilato)]

You can increase this max default value in index settings but you need to understand the risks associated with this.

You should though think about your users.

Why a user would randomly access to page 1020 (for example)? Would he do that on google or Qwant? I don't think so.

What is the use case?

[Adarsh_Bajpai]

Hey dadoonet,

Thanks for the reply, I tried increasing limit but it's slow down my cluster. Is there is any other way of getting these records?

We have a web application where we store users data in elasticsearch,
we have a listing page which shows all the records that the user has created with default pagination size of 100. he can increase pagination limit to 500 or 1000

So, if a user has created data more than 10000 which is quite usual in my case, the pages created will be greater than 100 (with pagination size of 100). To view, the previous record user can go to any page or can even click on the last page.

The user wouldn't click on page 1020 but can click on page 21 and if the page size is selected to 500, the elasticsearch will again throw an error.

So as you can see in pagination(image) user can click on any page even on page 3432 the last page.

[dadoonet (David Pilato)]

The question is "why a user would like to see the less relevant results?"

Page 20 is less relevant than page 1.
Do you go to page 20 when you search on Google?

I don't think you do. Why your users would have to do this?

After the fact it's slower: yes. That's one of the price to pay. The other is memory consumption.

[Adarsh_Bajpai]

So the page 20 is not less relevant is our case, suppose if a user searches for all the pin codes for a country and if a country has more than 10000 pin codes, then the listing page will contain more pages and user can manually visit any page.

[dadoonet (David Pilato)]

I'm sure there is a user story. You need to help your user to find the right information for him.
For this you can use faceted navigation pattern, you can sort by whatever field but leaving your user scrolling over pages does look user friendly to me.

That said, it can happen that the user explicitly ask for this. In which case you will have to pay the price as I mentioned before.

[Adarsh_Bajpai]

thanks for the help, but I think I had to look for other alternatives to elasticsearch.

[dadoonet (David Pilato)]

Sure but whatever the technical implementation you will have to think about this.

On elasticsearch side, you can always ask for next page with search after feature. Or scroll if you want to export the result set to your application or to the user.