# Inbuilt support for pagination? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 23, 2011, 4:15pm UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695 "2011-06-23T16:15:14Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Hari\_Shankar](https://avatars.discourse-cdn.com/v4/letter/h/ad7895/32.png) [@Hari\_Shankar](https://discuss.elastic.co/u/Hari_Shankar)\ **Post date:** [June 23, 2011, 4:15pm UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/1 "2011-06-23T16:15:14Z")
Hi, We are trying to use es as the data store for our UI. One of the major requirements for this is sorting and pagination of sorted data. Is there any inbuilt support in es for pagination? I tried scroll, but it seems it does not work for sorted data. e.g, I have 10 million records, which I want to sort on the basis of a particular (numeric) field, and I show 20 results/page. So if the user clicks on page 5, I'd like to show results 81-100 directly. Also, how does es do sorting on numeric fields? I am assuming it puts records into buckets of different ranges. (I am completely new to indexing/search). Also, is fetching the top 20 results of a set faster than fetching the top 1000, or fetching results 4001-4010, for example? Thanks, Hari ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [June 23, 2011, 7:22pm UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/2 "2011-06-23T19:22:03Z")
Scrolling should only be used in rare instances. From the HTTP API, size and from should do the trick for pagination: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Sort fields are pulled into memory which makes the sort operations quite fast. Not sure the exact method used, though. Also, my experience with pagination is that it is quite fast and I haven't noticed any performance degradation, even paginating beyond the 50,000th result. Best Regards, Paul On Jun 23, 10:15 am, Hari Shankar [shaan.h...@gmail.com](mailto:shaan.h...@gmail.com) wrote: > Hi, > > We are trying to use es as the data store for our UI. One of the major > requirements for this is sorting and pagination of sorted data. Is there any > inbuilt support in es for pagination? I tried scroll, but it seems it does > not work for sorted data. e.g, I have 10 million records, which I want to > sort on the basis of a particular (numeric) field, and I show 20 > results/page. So if the user clicks on page 5, I'd like to show results > 81-100 directly. > > Also, how does es do sorting on numeric fields? I am assuming it puts > records into buckets of different ranges. (I am completely new to > indexing/search). Also, is fetching the top 20 results of a set faster than > fetching the top 1000, or fetching results 4001-4010, for example? > > Thanks, > Hari ---
**Author:** ![Hari\_Shankar](https://avatars.discourse-cdn.com/v4/letter/h/ad7895/32.png) [@Hari\_Shankar](https://discuss.elastic.co/u/Hari_Shankar)\ **Post date:** [June 24, 2011, 5:21am UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/3 "2011-06-24T05:21:14Z")
But how would from+size work with sorting? The ids would not be sorted once we sort based on another column right? The way we are thinking right now is, for example, to go to the second page, to check the 20th value of the sorted field, and use this value in a "from" range query for the next request. But it would be cumbersome to do this correctly when the sorted field is not necessarily unique. Also, handling it when there are multiple sort fields will be cumbersome. Hari On Fri, Jun 24, 2011 at 12:52 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Scrolling should only be used in rare instances. > > From the HTTP API, size and from should do the trick for pagination: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/uri-request.html) > > Sort fields are pulled into memory which makes the sort operations > quite fast. Not sure the exact method used, though. Also, my > experience with pagination is that it is quite fast and I haven't > noticed any performance degradation, even paginating beyond the > 50,000th result. > > Best Regards, > Paul > > On Jun 23, 10:15 am, Hari Shankar [shaan.h...@gmail.com](mailto:shaan.h...@gmail.com) wrote: > > > Hi, > > > > We are trying to use es as the data store for our UI. One of the major > > requirements for this is sorting and pagination of sorted data. Is there > > any > > inbuilt support in es for pagination? I tried scroll, but it seems it > > does > > not work for sorted data. e.g, I have 10 million records, which I want to > > sort on the basis of a particular (numeric) field, and I show 20 > > results/page. So if the user clicks on page 5, I'd like to show results > > 81-100 directly. > > > > Also, how does es do sorting on numeric fields? I am assuming it puts > > records into buckets of different ranges. (I am completely new to > > indexing/search). Also, is fetching the top 20 results of a set faster > > than > > fetching the top 1000, or fetching results 4001-4010, for example? > > > > Thanks, > > Hari ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [June 24, 2011, 9:54am UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/4 "2011-06-24T09:54:40Z")
On Fri, 2011-06-24 at 10:51 +0530, Hari Shankar wrote: > But how would from+size work with sorting? The ids would not be sorted > once we sort based on another column right? The way we are thinking > right now is, for example, to go to the second page, to check the 20th > value of the sorted field, and use this value in a "from" range query > for the next request. But it would be cumbersome to do this correctly > when the sorted field is not necessarily unique. Also, handling it > when there are multiple sort fields will be cumbersome. Just to be clear, the from field takes a position, not an ID, so: page - from - size 1 0 10 2 10 10 3 20 10 50 490 10 And sort order is preserved even when the sort value is not unique. However, the number of docs that need to be processed in order to return (eg) page 50 is 500 \* no\_of\_shards = 2500 (assuming 5 primary shards). So you really don't want to offer to return page 5 million. Do like google and max out at 1,000 results. Who WANTS to see page 5 million anyway? If you need to retrieve all 5 million docs that match a query, eg to reindex or export them, then use a scrolled search with search\_type=scan - they won't be sorted, but it won't kill your ES server either 🙂 clint ---
**Author:** ![Hari\_Shankar](https://avatars.discourse-cdn.com/v4/letter/h/ad7895/32.png) [@Hari\_Shankar](https://discuss.elastic.co/u/Hari_Shankar)\ **Post date:** [June 24, 2011, 10:18am UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/5 "2011-06-24T10:18:15Z")
Ah, I was thinking from was based on id. Thanks, Hari On Fri, Jun 24, 2011 at 3:24 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote: > On Fri, 2011-06-24 at 10:51 +0530, Hari Shankar wrote: > > > But how would from+size work with sorting? The ids would not be sorted > > once we sort based on another column right? The way we are thinking > > right now is, for example, to go to the second page, to check the 20th > > value of the sorted field, and use this value in a "from" range query > > for the next request. But it would be cumbersome to do this correctly > > when the sorted field is not necessarily unique. Also, handling it > > when there are multiple sort fields will be cumbersome. > > Just to be clear, the from field takes a position, not an ID, so: > > page - from - size > 1 0 10 > 2 10 10 > 3 20 10 > 50 490 10 > > And sort order is preserved even when the sort value is not unique. > > However, the number of docs that need to be processed in order to return > (eg) page 50 is 500 \* no\_of\_shards = 2500 (assuming 5 primary shards). > So you really don't want to offer to return page 5 million. > > Do like google and max out at 1,000 results. Who WANTS to see page 5 > million anyway? > > If you need to retrieve all 5 million docs that match a query, eg to > reindex or export them, then use a scrolled search with search\_type=scan > > - they won't be sorted, but it won't kill your ES server either 🙂 > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 24, 2011, 12:12pm UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/6 "2011-06-24T12:12:38Z")
Also note that elasticsearch takes special care to try and optimize "long tail" pagination (though there is a limit, of course). The "query\_then\_fetch" type makes sure to only fetch doc ids from all shards to do the pagination calculation, and only them goes and fetch the relevant docs needed. On Friday, June 24, 2011 at 1:18 PM, Hari Shankar wrote: > Ah, I was thinking from was based on id. > > Thanks, > Hari > > On Fri, Jun 24, 2011 at 3:24 PM, Clinton Gormley \<[clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) ([mailto:clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk))\> wrote: > > > On Fri, 2011-06-24 at 10:51 +0530, Hari Shankar wrote: > > > > > But how would from+size work with sorting? The ids would not be sorted > > > once we sort based on another column right? The way we are thinking > > > right now is, for example, to go to the second page, to check the 20th > > > value of the sorted field, and use this value in a "from" range query > > > for the next request. But it would be cumbersome to do this correctly > > > when the sorted field is not necessarily unique. Also, handling it > > > when there are multiple sort fields will be cumbersome. > > > > Just to be clear, the from field takes a position, not an ID, so: > > > > page - from - size > > 1 0 10 > > 2 10 10 > > 3 20 10 > > 50 490 10 > > > > And sort order is preserved even when the sort value is not unique. > > > > However, the number of docs that need to be processed in order to return > > (eg) page 50 is 500 \* no\_of\_shards = 2500 (assuming 5 primary shards). > > So you really don't want to offer to return page 5 million. > > > > Do like google and max out at 1,000 results. Who WANTS to see page 5 > > million anyway? > > > > If you need to retrieve all 5 million docs that match a query, eg to > > reindex or export them, then use a scrolled search with search\_type=scan > > > > - they won't be sorted, but it won't kill your ES server either 🙂 > > > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:02am UTC](https://discuss.elastic.co/t/inbuilt-support-for-pagination/4695/7 "2017-07-06T04:02:42Z")