# Fetch data from a middle of a big stack using searchAfter( jump to a specific page)

**URL:** <https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256>\
**Category:** Elasticsearch\
**Tags:** point-in-time\
**Created:** [September 27, 2021, 4:23pm UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256 "2021-09-27T16:23:48Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Fat](https://avatars.discourse-cdn.com/v4/letter/f/dfb087/32.png) [@Fat](https://discuss.elastic.co/u/Fat)\
**Post date:** [September 27, 2021, 4:23pm UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/1 "2021-09-27T16:23:48Z")

</div>

Hi All  
I have a large data set around 25million records  
I am using searchAfter with PointInTime to walk through the data  
My question is there a way where I can skip records over the limit of 10000

> index.max\_result\_window

and start picking the records for example from 100,000 up to 105,000

right now I am sending multiple requests to Elasticsearch until I reach the desired point but it is not efficient and it is consuming a lot of time

Here is how I did it :  
I calculated how many pages I needed to do the pagination.  
Then the user will send a request with page number i.e number 3. So in this case only when I reach the desired page I will set the source to true.  
this I best I managed to do to improve the performance and reduce the response size for none required pages

```auto
 int numberOfPages = Pagination.GetTotalPages(totalCount, _size);

 var pitResponse = await _esClient.OpenPointInTimeAsync(content._index, p => p.KeepAlive("2m"));

            if (pitResponse.IsValid)
            {
                IEnumerable<object> lastHit = null;

                    for (int round = 0; round < numberOfPages; round++)
                    {
                        bool fetchSource = round == requiredPage;
                        var response = await _esClient.SearchAsync<ProductionDataItem>(s => s
                            .Index(content._index)
                            .Size(10000)
                            .Source(fetchSource)
                            .Query(query)
                            .PointInTime(pitResponse.Id)
                            .Sort(srt => {
                                if (content.Sort == 1) { srt.Ascending(sortBy); }
                                else { srt.Descending(sortBy); }
                                return srt; })
                            .SearchAfter(lastHit)
                        );

                        if (fetchSource)
                        {
                           itemsList.AddRange(response.Documents.ToList());
                            break;
                        }
                        lastHit = response.Hits.Last().Sorts;
                    }
                }
                //Closing PIT
                await _esClient.ClosePointInTimeAsync(p => p.Id(pitResponse.Id));

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 28, 2021, 4:40am UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/2 "2021-09-28T04:40:12Z")

</div>

Welcome to our community! 😃

I'm not super familiar with PIT, but can you show us the request that you are making?

---

<div class="post-metadata">

**Author:** ![Fat](https://avatars.discourse-cdn.com/v4/letter/f/dfb087/32.png) [@Fat](https://discuss.elastic.co/u/Fat)\
**Post date:** [September 28, 2021, 8:41am UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/3 "2021-09-28T08:41:23Z")

</div>

Thanks for replay I updated the post

---

<div class="post-metadata">

**Author:** ![Fat](https://avatars.discourse-cdn.com/v4/letter/f/dfb087/32.png) [@Fat](https://discuss.elastic.co/u/Fat)\
**Post date:** [September 30, 2021, 10:11am UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/4 "2021-09-30T10:11:36Z")

</div>

Any feedback ?

---

<div class="post-metadata">

**Author:** ![Fat](https://avatars.discourse-cdn.com/v4/letter/f/dfb087/32.png) [@Fat](https://discuss.elastic.co/u/Fat)\
**Post date:** [October 1, 2021, 8:18am UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/5 "2021-10-01T08:18:13Z")

</div>

I think the best way to do it, is how I did it

by keeping scrolling via Point in time and only loading the result when the desired page is reached by using the .source(bool)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 29, 2021, 8:19am UTC](https://discuss.elastic.co/t/fetch-data-from-a-middle-of-a-big-stack-using-searchafter-jump-to-a-specific-page/285256/6 "2021-10-29T08:19:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.