# ES Java matchAllQuery paging size limit

**URL:** <https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337>\
**Category:** Elasticsearch\
**Created:** [March 24, 2016, 11:23am UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337 "2016-03-24T11:23:04Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ajmariella](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ajmariella](https://discuss.elastic.co/u/ajmariella)\
**Post date:** [March 24, 2016, 11:23am UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/1 "2016-03-24T11:23:04Z")

</div>

I'm using the java API to do a matchAllQuery() including .setFetchSource for some particular fields to get returned.  
I am also using .setFrom and .setSize. WHen I get my initial results back, I get totalHits = 4,9123,345.  
But when I try paging through the results, when I get to 10001 I get a NullPointer Error on the response.  
Am I doing something wrong ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 24, 2016, 11:39am UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/2 "2016-03-24T11:39:44Z")

</div>

We added recently a limit for deep pagination.  
Is it something you can reproduce? Potentially with a pure CURL script?

If you can reproduce it, please open an issue with that.

If not, can you share here more code? May be you have a NPE in your own code?

BTW, for deep pagination, use scroll API instead.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 24, 2016, 11:40am UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/3 "2016-03-24T11:40:32Z")

</div>

I forgot. A full stacktrace would help a lot.

---

<div class="post-metadata">

**Author:** ![ajmariella](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ajmariella](https://discuss.elastic.co/u/ajmariella)\
**Post date:** [March 24, 2016, 12:00pm UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/4 "2016-03-24T12:00:15Z")

</div>

I added .setScroll and I didn't get the error again. But I keep getting results no matter what page of results I add to the query. Based on the page number, I set .setFrom((page -1) \* BULK\_MAX\_RESULTS) where BULK\_MAX\_RESULTS = 10000

How do I know when I get to the end of the data ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 24, 2016, 1:48pm UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/5 "2016-03-24T13:48:53Z")

</div>

Read this: [https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/java-search-scrolling.html)

---

<div class="post-metadata">

**Author:** ![ajmariella](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ajmariella](https://discuss.elastic.co/u/ajmariella)\
**Post date:** [March 24, 2016, 2:12pm UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/6 "2016-03-24T14:12:06Z")

</div>

I'm writing a web service to retrieve the data, so I can't send all the data at once, I'm relying on the user to input a page number in order to retrieve data. Is there another way to retrieve the data besides scrolling ?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [March 24, 2016, 2:30pm UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/7 "2016-03-24T14:30:49Z")

</div>

Deep scrolling is always bad. It works in relational databases but poorly. In Elasticsearch it works even more poorly. That is why we don't allow it by default.

One way to work around this is to sort by something consistent and have the user ask for all results after their last one. Like, if you are paging through results by some auto incrementing id then have them ask for documents \>= the last one. Or if you are doing date then sort by date, id and have them ask for stuff after the last date, id pair. The advantage of this way of doing things is that you get to use the index to skip all the documents rather than visiting them all. The disadvantage is that you have to sort. If you were sorting any way, well, then no big deal. If not, well, I dunno what is right then.

---

<div class="post-metadata">

**Author:** ![ajmariella](https://avatars.discourse-cdn.com/v4/letter/a/9dc877/32.png) [@ajmariella](https://discuss.elastic.co/u/ajmariella)\
**Post date:** [March 25, 2016, 11:30am UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/8 "2016-03-25T11:30:45Z")

</div>

Is there a way with the Java API to only get unique values from a particular field ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:05pm UTC](https://discuss.elastic.co/t/es-java-matchallquery-paging-size-limit/45337/9 "2017-07-05T23:05:15Z")

</div>


