# Elastic search response capped to 10k records

**URL:** <https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192>\
**Category:** Elasticsearch\
**Created:** [September 12, 2022, 12:05pm UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192 "2022-09-12T12:05:30Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deepanshu\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepanshu_rai/32/100222_2.png) [@Deepanshu\_Rai](https://discuss.elastic.co/u/Deepanshu_Rai)\
**Post date:** [September 12, 2022, 12:05pm UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/1 "2022-09-12T12:05:30Z")

</div>

I am trying to query my index which has more then 500k records but i am only able to extract 10k records at a time. now i understand this has to do with performance of the application but how can i do bulk extract?  
with records more than 50k, i don't think option of "size" and "after " will be correct way to go ahead.  
current ES version we are using is 7.10.2

```auto
"version" : {
    "number" : "7.10.2",
    "build_flavor" : "oss"
}

```

I tried to use the concept of PIT ID for this purpose but i am not sure if this version of ES supports PIT id because when i used dev tool with PIT id concept, it didn't work.

Please help how can i do bulk get or any pagination approach for more then 500k records.

thanks.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 12, 2022, 12:41pm UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/2 "2022-09-12T12:41:25Z")

</div>

The Point in Time API is not available in the OSS version, you need to use the Scroll API.

Check the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/scroll-api.html) and this example on how to paginate using the [scroll api](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/paginate-search-results.html#scroll-search-results).

If you are using a client, like the [Python client](https://elasticsearch-py.readthedocs.io/en/7.x/helpers.html#scan), there is a helper called `scan` to help you do queries like this.

---

<div class="post-metadata">

**Author:** ![Deepanshu\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/deepanshu_rai/32/100222_2.png) [@Deepanshu\_Rai](https://discuss.elastic.co/u/Deepanshu_Rai)\
**Post date:** [September 14, 2022, 6:54am UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/3 "2022-09-14T06:54:21Z")

</div>

@leandrojmp thanks for reply. but as mentioned in documentation, scroll is not recommended for deep pagination for more than 10k records. In my case, i have more then 500k records which i need to extract.  
I am not using any client, rather making direct http rest calls to extract data from ES but as mentioned earlier, not able to get more than 10k at a time.  
below is my sample request.

```auto
curl -X POST "localhost/index/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": {
    "match": {
      "person.name": "ABC"
    }
  },
  "fields": [
    "person.age"
  ],
  "_source": false
}
'

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 14, 2022, 7:38am UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/4 "2022-09-14T07:38:05Z")

</div>

> [@Deepanshu\_Rai](#):
>
> scroll is not recommended for deep pagination for more than 10k records.

The documentation points out that you should use search\_after together with PIT. As you are using the OSS version where this is not available, using the scroll API for deep pagination is still the recommended option.

My recommendation would however be to switch to the default distribution and upgrade at to the latest 7.17 release.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [September 14, 2022, 12:07pm UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/5 "2022-09-14T12:07:16Z")

</div>

As Christian already answered, this recommendation is based in the Elastic distribution of Elasticsearch using at least the basic license.

You are using the Open Source distribution which does not have the PIT feature, so the recommendation in this case is to use the scroll API.

You can still use the scroll API without using a client, just check in the documentation on how to do it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2022, 12:07pm UTC](https://discuss.elastic.co/t/elastic-search-response-capped-to-10k-records/314192/6 "2022-10-12T12:07:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.