# Speeding up deep pagination for large ids query

**URL:** <https://discuss.elastic.co/t/speeding-up-deep-pagination-for-large-ids-query/351636>\
**Category:** Elasticsearch\
**Created:** [January 23, 2024, 2:45pm UTC](https://discuss.elastic.co/t/speeding-up-deep-pagination-for-large-ids-query/351636 "2024-01-23T14:45:08Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dsc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dsc/32/100102_2.png) [@dsc](https://discuss.elastic.co/u/dsc)\
**Post date:** [January 23, 2024, 2:45pm UTC](https://discuss.elastic.co/t/speeding-up-deep-pagination-for-large-ids-query/351636/1 "2024-01-23T14:45:08Z")

</div>

I've got an Elasticsearch index with ~100M documents, and typically need to search within a subset of them using an [IDs query](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ids-query.html) combined with other search terms/filters.

These subsets of IDs are dynamic and come from an external system, so aren't something we can keep indexed in Elasticsearch.

When performing pagination using the [Point in Time](https://www.elastic.co/guide/en/elasticsearch/reference/current/point-in-time-api.html) functionality, fetching each subsequent set of results (using `size=10000` and `search_after`) gets extremely slow when using a large list of IDs.

For example, using a list of ~200,000 IDs - fetching each set of 10,000 results using PIT and search\_after takes ~20 seconds. So fetching 100,000 documents this way takes over 3 minutes.

Are there any good ways to speed this up? I'd have expected the initial query containing such a large number of IDs to be slow, but it appears that the cost for using such a larger number of IDs is paid every time the next set of results if fetched, rather than just once.

For info, my search looks something like:

```auto
{
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {"query": "some query"}
        },
        {
          "ids": {"values": [... ~200k integer IDs ...]}
        }
      ]
    }
  }
  "_source": False,
  "fields": ["title"],
  "sort": "_score",
  "size": 10000,
  "track_total_hits": true,
  "track_scores": false,
  "pit": {
    "id": "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", 
    "keep_alive": "1m"
  },
  "sort": "title",
  "search_after": [...]                       
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2024, 2:45pm UTC](https://discuss.elastic.co/t/speeding-up-deep-pagination-for-large-ids-query/351636/2 "2024-02-20T14:45:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.