# Skipped records with Search/SearchAfter query

**URL:** <https://discuss.elastic.co/t/skipped-records-with-search-searchafter-query/314205>\
**Category:** Elasticsearch\
**Created:** [September 12, 2022, 4:04pm UTC](https://discuss.elastic.co/t/skipped-records-with-search-searchafter-query/314205 "2022-09-12T16:04:19Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![kuhu.88](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kuhu.88](https://discuss.elastic.co/u/kuhu.88)\
**Post date:** [September 12, 2022, 4:04pm UTC](https://discuss.elastic.co/t/skipped-records-with-search-searchafter-query/314205/1 "2022-09-12T16:04:19Z")

</div>

Hello folks,

Using a non-unique key for sort (in this example - timestamp) can skip over records in a subsequent request when the 10,000 is reached on a previous run and timestamps carry on beyond that cutoff.

Take the example that for one of the results, half of the results are for timestamp=`t1` and half are with timestamp=`t2`. The next 10,000 records contains some (say 500 entries) with timestamp=`t2`. Based on the current implementation, the value of the `sort` array for the next query would be `t2`. Elasticsearch in the next response will skip over the 500 entries in the next chunk. This can cause much less results coming through compared to the expected count. The expected counts based on the `_count` API in ES and the results of `scroll` which match. The search query return an order of magnitude less for some of the test runs.

An example query --

```auto
curl -X GET "localhost:9200/indexname*/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 10000,
  "_source": ["abc.123", "abc.124"],
  "query": {
      "range": {
      "@timestamp": {
        "gte": "now-1d/d",
        "lt": "now/d"
      }
    }
  },
  "sort": [
    {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}}
  ]
}
'

```

The next query would be something like:

```auto
curl -X GET "localhost:9200/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size": 10000,
  "query": {
      "range": {
      "@timestamp": {
        "gte": "now-4h/h",
        "lt": "now/h"
      }
    }
  },
  "sort": [
    {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}}
  ]

 "search_after": [                                
   "2022-06-22T03:00:28.000Z"
 ]
}
'

```

Using PIT does not mitigate this issue of needing to go through every record that matches as it is used only as a tie-breaker and timestamp seems to win which means the skipping will occur. Appreciate any ideas and thoughts on next steps.

ES version: 7.16.2

---

<div class="post-metadata">

**Author:** ![kuhu.88](https://avatars.discourse-cdn.com/v4/letter/k/e9bcb4/32.png) [@kuhu.88](https://discuss.elastic.co/u/kuhu.88)\
**Post date:** [October 4, 2022, 9:54pm UTC](https://discuss.elastic.co/t/skipped-records-with-search-searchafter-query/314205/2 "2022-10-04T21:54:41Z")

</div>

Any updates on this?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 1, 2022, 9:55pm UTC](https://discuss.elastic.co/t/skipped-records-with-search-searchafter-query/314205/3 "2022-11-01T21:55:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.