Is ES capable of doing pagination?

Elastic Stack Elasticsearch
1

Hi,

I am fairly new to ES. I was wondering if ES is capable of doing the
pagination? My use case is if the query result is huge, can ES return a
specific amount of data as request? For example, I might only need records
100-200 from the overall result.

Regards
Boris

2

You should look at this
http://www.elasticsearch.org/guide/reference/api/search/request-body.html and
more precisely the from and size parameters.

3

Thanks for the information, it is very useful. One more question, does the
data returned from ES contain the total result count when the request
contains the size parameters?

Thanks in advance
Boris

On Thu, Oct 6, 2011 at 5:14 PM, Benjamin Devèze
benjamin.deveze@gmail.comwrote:

You should look at this
Elasticsearch Platform — Find real-time answers at scale | Elastic and
more precisely the from and size parameters.

4

Yeah you will be happy it does contain the total result count.

5

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

6

Thanks for the information, I will look into it.

On Fri, Oct 7, 2011 at 1:19 PM, Jamshid jamshid69@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

7

scan search type is limited:

Note scan search type does not support sorting (either on score or
a field) or faceting.

You can just use a timestamp for every document and filter newer
documents away when doing pagination. (same procedure like twitter
handles it)

On 7 Okt., 07:19, Jamshid jamshi...@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

8

Interesting, I hadn't thought of that. So the timestamp would be the
"create date" of the document? Not sure how filtering would work when
updating or deleting documents, would elasticsearch keep the updated
record in the same "place" while paginating? But I don't see a
solution if documents are deleted while paging.
--Jamshid

On Oct 12, 10:04 am, Karussell tableyourt...@googlemail.com wrote:

scan search type is limited:

Note scan search type does not support sorting (either on score or
a field) or faceting.

You can just use a timestamp for every document and filter newer
documents away when doing pagination. (same procedure like twitter
handles it)

On 7 Okt., 07:19, Jamshid jamshi...@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

9

Scan search type is a point in time search, when its executed. You won't see
changes (either deletions or new docs) after its first execution.

On Thu, Oct 13, 2011 at 9:07 PM, Jamshid jamshid69@gmail.com wrote:

Interesting, I hadn't thought of that. So the timestamp would be the
"create date" of the document? Not sure how filtering would work when
updating or deleting documents, would elasticsearch keep the updated
record in the same "place" while paginating? But I don't see a
solution if documents are deleted while paging.
--Jamshid

On Oct 12, 10:04 am, Karussell tableyourt...@googlemail.com wrote:

scan search type is limited:

Note scan search type does not support sorting (either on score or
a field) or faceting.

You can just use a timestamp for every document and filter newer
documents away when doing pagination. (same procedure like twitter
handles it)

On 7 Okt., 07:19, Jamshid jamshi...@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (
Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

10

Right, I was referring to Karussell's suggestion of using a timestamp
to allow regular (non-"scan") searches to return a consistent set of
results.

I want sorted results (rules out "scan") and I want to be able to page
through the results without missing documents or seeing duplicates.

Any suggestions, has this kind of request come up before?

Thanks,
Jamshid

On Oct 14, 7:23 am, Shay Banon kim...@gmail.com wrote:

Scan search type is a point in time search, when its executed. You won't see
changes (either deletions or new docs) after its first execution.

On Thu, Oct 13, 2011 at 9:07 PM, Jamshid jamshi...@gmail.com wrote:

Interesting, I hadn't thought of that. So the timestamp would be the
"create date" of the document? Not sure how filtering would work when
updating or deleting documents, would elasticsearch keep the updated
record in the same "place" while paginating? But I don't see a
solution if documents are deleted while paging.
--Jamshid

On Oct 12, 10:04 am, Karussell tableyourt...@googlemail.com wrote:

scan search type is limited:

Note scan search type does not support sorting (either on score or
a field) or faceting.

You can just use a timestamp for every document and filter newer
documents away when doing pagination. (same procedure like twitter
handles it)

On 7 Okt., 07:19, Jamshid jamshi...@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (
Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com wrote:

Yeah you will be happy it does contain the total result count.

11

You can use scrolling with search_type (like the default, query_then_fetch),
which will guarantee you won't see duplicates or changed data, but, it gets
more and more expensive as you scroll through the results.

On Fri, Oct 14, 2011 at 8:42 PM, Jamshid jamshid69@gmail.com wrote:

Right, I was referring to Karussell's suggestion of using a timestamp
to allow regular (non-"scan") searches to return a consistent set of
results.

I want sorted results (rules out "scan") and I want to be able to page
through the results without missing documents or seeing duplicates.

Any suggestions, has this kind of request come up before?

Thanks,
Jamshid

On Oct 14, 7:23 am, Shay Banon kim...@gmail.com wrote:

Scan search type is a point in time search, when its executed. You won't
see
changes (either deletions or new docs) after its first execution.

On Thu, Oct 13, 2011 at 9:07 PM, Jamshid jamshi...@gmail.com wrote:

Interesting, I hadn't thought of that. So the timestamp would be the
"create date" of the document? Not sure how filtering would work when
updating or deleting documents, would elasticsearch keep the updated
record in the same "place" while paginating? But I don't see a
solution if documents are deleted while paging.
--Jamshid

On Oct 12, 10:04 am, Karussell tableyourt...@googlemail.com wrote:

scan search type is limited:

Note scan search type does not support sorting (either on score
or
a field) or faceting.

You can just use a timestamp for every document and filter newer
documents away when doing pagination. (same procedure like twitter
handles it)

On 7 Okt., 07:19, Jamshid jamshi...@gmail.com wrote:

One thing to keep in mind is that if you are continuously indexing,
don't expect to be able to page through the results without missing
some items or seeing duplicates.

You can use the "scan" search type (
Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/api/search/search-type.html) instead if you want a
consistent result set that you can page through, but unfortunately
you
will not be able to sort the results when using "scan".

--Jamshid

On Oct 6, 9:34 am, Benjamin Devèze benjamin.dev...@gmail.com
wrote:

Yeah you will be happy it does contain the total result count.

12

So you can use elastic search queries to page through a large dataset using an offset and limit.

If I have 10 million objects to page through, and want them 1000 at a time
(say) will the search be very slow when I ask for "The next 1000 items starting with the 9 millionth item" ?

13

Yes, it will get slower the farther through the result set you get. It
needs to perform a query for results as far as the (offset+limit) doc, then
select/return the 1000 docs you wanted in the page.

The proposed alternative which does not incur this same performance hit is
to use a scan/scroll query, which keeps a result set open for a period of
time as you page over it. See the section on scan
herehttp://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-search-type.html
.

On Tue, Oct 22, 2013 at 8:35 AM, rodders7884 rod.french@gmail.com wrote:

So you can use Elasticsearch queries to page through a large dataset using
an offset and limit.

If I have 10 million objects to page through, and want them 1000 at a time
(say) will the search be very slow when I ask for "The next 1000 items
starting with the 9 millionth item" ?

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Is-ES-capable-of-doing-pagination-tp3399087p4042984.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

14

can somebody suggest..

Is it possible to jump over Page Nos [forward/backward]while using Scan/Scroll for Pagination?.. Or any other effective way available for pagination?.

15

Hi Prasath,

Scan and scroll can only move forward. If you want to have previous/next
buttons, you would typically run the query once again with different values
of from and size.

On Tue, Jul 22, 2014 at 3:16 PM, PrasathRajan prasanth.sunrise@gmail.com
wrote:

can somebody suggest..

Is it possible to jump over Page Nos [forward/backward]while using
Scan/Scroll for Pagination?.. Or any other effective way available for
pagination?.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Is-ES-capable-of-doing-pagination-tp3399087p4060343.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1406034965952-4060343.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6y-hQudM4utSi-JJakdMrkJLXN90MwRa8Tu9xpuhnbyA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

16

Scan/scroll is also not for exposing to "web scale" users. Fine for tens
of users, not for millions. There is non-trivial cost on the cluster
during scan/scroll.

For the most part we just use from and size. There is a setting called
preference that might be worth looking at if you expect lots of scrolling.

Don't allow super deep scrolling - there are some portions of query
execution that end up having to do from + size amounts of work (O(n*log(n)
of it). And they need O(from + size) of memory too. So going too deep can
take up tons of memory.

Nik

On Wed, Jul 23, 2014 at 9:36 AM, Adrien Grand <
adrien.grand@elasticsearch.com> wrote:

Hi Prasath,

Scan and scroll can only move forward. If you want to have previous/next
buttons, you would typically run the query once again with different values
of from and size.

On Tue, Jul 22, 2014 at 3:16 PM, PrasathRajan prasanth.sunrise@gmail.com
wrote:

can somebody suggest..

Is it possible to jump over Page Nos [forward/backward]while using
Scan/Scroll for Pagination?.. Or any other effective way available for
pagination?.

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Is-ES-capable-of-doing-pagination-tp3399087p4060343.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1406034965952-4060343.post%40n3.nabble.com
.
For more options, visit https://groups.google.com/d/optout.

--
Adrien Grand

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6y-hQudM4utSi-JJakdMrkJLXN90MwRa8Tu9xpuhnbyA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAL6Z4j6y-hQudM4utSi-JJakdMrkJLXN90MwRa8Tu9xpuhnbyA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2ygvWv0%2BuaQ1UNpoJwksu_sRLPdUw3cz5-gsf-uDEF%3Dw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.