# Search\_after queries are not performing as expected

**URL:** <https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174>\
**Category:** Elasticsearch\
**Created:** [November 15, 2021, 8:54am UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174 "2021-11-15T08:54:27Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![manuel.pancorbo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.pancorbo/32/80928_2.png) [@manuel.pancorbo](https://discuss.elastic.co/u/manuel.pancorbo)\
**Post date:** [November 15, 2021, 8:54am UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/1 "2021-11-15T08:54:27Z")

</div>

Hi all!

I'm working on building a searcher for a site. This site generates a lot of writes (new or modified documents) so we need to use some mechanism to ensure pagination consistency by using certain criteria (freshness or price in our cases).

We began to use the search\_after parameter for solving our issues. This works fine until we realized that our query slow log started to have a lot of entries. At some moments this lead us to high CPU usage and high search latency, which degraded our service.

After a deeper look, we realized that all the slow queries look like the following one:

```json
{
  "size": 30,
  "sort": [
    {
      "sortDate": {
        "order": "desc"
      },
      "id": {
        "order": "desc"
      }
    }
  ],
  "search_after": [
    1636109510000,
    {any_id}
  ],
  "track_total_hits": 10000
}

```

All of these log entries show us queries without filters or search terms and use the search\_after parameter. This kind of query is taking ~400ms to be resolved. The issue is that its equivalent using a simple from/size pagination is taking just ~20ms.

So, why ES is taking more time/resources to resolve the query with "search\_after" than its equivalent with from/size?

We realized that this issue is only happening when searching without filters (our index has ~15M of documents), by filtering for some criteria that narrow down the dataset this issue disappears.

Following a suggestion, we discovered index sorting. After applying it, we reduce search latency dramatically for both queries (with or without search\_after), but just for one of our sorting criteria.

So, our questions are:

Why some queries using search\_after are performing so bad compared with their from/size equivalents?  
If the solution is to help index by adding index sorting, what can we do to improve query times for different search criteria?

Some examples of the queries **before applying the index sorting** (forget that localhost, I'm just using a proxy for reaching the production cluster):

 ![search_after_pagination](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f9af12f5f41fe0f2db3cd639b12cce2c3a0c9e0.png)  
 ![classic_pagination_query](https://us1.discourse-cdn.com/elastic/original/3X/4/7/470aa8277232256dcdad833ab96f09802484a537.png)

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [November 15, 2021, 11:10pm UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/2 "2021-11-15T23:10:44Z")

</div>

What Elasticsearch version are you using?

Can you please post an output of [Profile API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html) for this slow sort query with `search_after` and also an equivalent from/size query that is much faster?

---

<div class="post-metadata">

**Author:** ![manuel.pancorbo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.pancorbo/32/80928_2.png) [@manuel.pancorbo](https://discuss.elastic.co/u/manuel.pancorbo)\
**Post date:** [November 16, 2021, 8:59am UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/3 "2021-11-16T08:59:50Z")

</div>

Hi Mayya! Thanks for answering,

We are using 7.10.

Profile of a from/size query:

```json
{
  "took": 56,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  },
  "profile": {
    "shards": [
      {
        "id": "[2ZdWfwd_T6ubj-jNQsivNA][ads-202111121136][0]",
        "searches": [
          {
            "query": [
              {
                "type": "BooleanQuery",
                "description": "#*:* LongDistanceFeatureQuery(field=,origin=999999999,pivotDistance=499999999)",
                "time_in_nanos": 20188168,
                "breakdown": {
                  "set_min_competitive_score_count": 257,
                  "match_count": 0,
                  "shallow_advance_count": 0,
                  "set_min_competitive_score": 12819400,
                  "next_doc": 3998602,
                  "match": 0,
                  "next_doc_count": 5543,
                  "score_count": 1425,
                  "compute_max_score_count": 0,
                  "compute_max_score": 0,
                  "advance": 334259,
                  "advance_count": 27,
                  "score": 1192773,
                  "build_scorer_count": 54,
                  "create_weight": 363068,
                  "shallow_advance": 0,
                  "create_weight_count": 1,
                  "build_scorer": 1480066
                },
                "children": [
                  {
                    "type": "MatchAllDocsQuery",
                    "description": "*:*",
                    "time_in_nanos": 1028258,
                    "breakdown": {
                      "set_min_competitive_score_count": 0,
                      "match_count": 0,
                      "shallow_advance_count": 0,
                      "set_min_competitive_score": 0,
                      "next_doc": 0,
                      "match": 0,
                      "next_doc_count": 0,
                      "score_count": 0,
                      "compute_max_score_count": 0,
                      "compute_max_score": 0,
                      "advance": 936037,
                      "advance_count": 9087,
                      "score": 0,
                      "build_scorer_count": 81,
                      "create_weight": 4463,
                      "shallow_advance": 0,
                      "create_weight_count": 1,
                      "build_scorer": 87758
                    }
                  },
                  {
                    "type": "LongDistanceFeatureQuery",
                    "description": "LongDistanceFeatureQuery(field=,origin=999999999,pivotDistance=499999999)",
                    "time_in_nanos": 13833607,
                    "breakdown": {
                      "set_min_competitive_score_count": 257,
                      "match_count": 0,
                      "shallow_advance_count": 81,
                      "set_min_competitive_score": 12529039,
                      "next_doc": 0,
                      "match": 0,
                      "next_doc_count": 0,
                      "score_count": 1425,
                      "compute_max_score_count": 54,
                      "compute_max_score": 7509,
                      "advance": 676901,
                      "advance_count": 5395,
                      "score": 501172,
                      "build_scorer_count": 54,
                      "create_weight": 1084,
                      "shallow_advance": 11638,
                      "create_weight_count": 1,
                      "build_scorer": 106264
                    }
                  }
                ]
              }
            ],
            "rewrite_time": 36734,
            "collector": [
              {
                "name": "SimpleFieldCollector",
                "reason": "search_top_hits",
                "time_in_nanos": 15452992
              }
            ]
          }
        ],
        "aggregations": []
      },
      {
        "id": "[W5AN-Ap6R9iy0R9OjMuHUg][ads-202111121136][1]",
        "searches": [
          {
            "query": [
              {
                "type": "BooleanQuery",
                "description": "#*:* LongDistanceFeatureQuery(field=,origin=999999999,pivotDistance=499999999)",
                "time_in_nanos": 21845509,
                "breakdown": {
                  "set_min_competitive_score_count": 289,
                  "match_count": 0,
                  "shallow_advance_count": 0,
                  "set_min_competitive_score": 14510762,
                  "next_doc": 3127208,
                  "match": 0,
                  "next_doc_count": 6003,
                  "score_count": 3342,
                  "compute_max_score_count": 0,
                  "compute_max_score": 0,
                  "advance": 225705,
                  "advance_count": 23,
                  "score": 1783918,
                  "build_scorer_count": 46,
                  "create_weight": 341011,
                  "shallow_advance": 0,
                  "create_weight_count": 1,
                  "build_scorer": 1856905
                },
                "children": [
                  {
                    "type": "MatchAllDocsQuery",
                    "description": "*:*",
                    "time_in_nanos": 747234,
                    "breakdown": {
                      "set_min_competitive_score_count": 0,
                      "match_count": 0,
                      "shallow_advance_count": 0,
                      "set_min_competitive_score": 0,
                      "next_doc": 0,
                      "match": 0,
                      "next_doc_count": 0,
                      "score_count": 0,
                      "compute_max_score_count": 0,
                      "compute_max_score": 0,
                      "advance": 651840,
                      "advance_count": 9324,
                      "score": 0,
                      "build_scorer_count": 69,
                      "create_weight": 4266,
                      "shallow_advance": 0,
                      "create_weight_count": 1,
                      "build_scorer": 91128
                    }
                  },
                  {
                    "type": "LongDistanceFeatureQuery",
                    "description": "LongDistanceFeatureQuery(field=,origin=999999999,pivotDistance=499999999)",
                    "time_in_nanos": 15461591,
                    "breakdown": {
                      "set_min_competitive_score_count": 289,
                      "match_count": 0,
                      "shallow_advance_count": 69,
                      "set_min_competitive_score": 13920752,
                      "next_doc": 0,
                      "match": 0,
                      "next_doc_count": 0,
                      "score_count": 3342,
                      "compute_max_score_count": 46,
                      "compute_max_score": 7272,
                      "advance": 673629,
                      "advance_count": 5710,
                      "score": 750093,
                      "build_scorer_count": 46,
                      "create_weight": 1034,
                      "shallow_advance": 11370,
                      "create_weight_count": 1,
                      "build_scorer": 97441
                    }
                  }
                ]
              }
            ],
            "rewrite_time": 34905,
            "collector": [
              {
                "name": "SimpleFieldCollector",
                "reason": "search_top_hits",
                "time_in_nanos": 19567276
              }
            ]
          }
        ],
        "aggregations": []
      }
    ]
  }
}

```

Profile of the equivalent query using search\_after:

```auto
{
  "took": 977,
  "timed_out": false,
  "_shards": {
    "total": 2,
    "successful": 2,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 10000,
      "relation": "gte"
    },
    "max_score": null,
    "hits": []
  },
  "profile": {
    "shards": [
      {
        "id": "[OXyAsvfDQqm5f6kWJze12g][ads-202111121136][0]",
        "searches": [
          {
            "query": [
              {
                "type": "MatchAllDocsQuery",
                "description": "*:*",
                "time_in_nanos": 335185552,
                "breakdown": {
                  "set_min_competitive_score_count": 0,
                  "match_count": 0,
                  "shallow_advance_count": 0,
                  "set_min_competitive_score": 0,
                  "next_doc": 334529838,
                  "match": 0,
                  "next_doc_count": 9621019,
                  "score_count": 0,
                  "compute_max_score_count": 0,
                  "compute_max_score": 0,
                  "advance": 9846,
                  "advance_count": 23,
                  "score": 0,
                  "build_scorer_count": 46,
                  "create_weight": 8788,
                  "shallow_advance": 0,
                  "create_weight_count": 1,
                  "build_scorer": 637080
                }
              }
            ],
            "rewrite_time": 21956,
            "collector": [
              {
                "name": "PagingFieldCollector",
                "reason": "search_top_hits",
                "time_in_nanos": 531797539
              }
            ]
          }
        ],
        "aggregations": []
      },
      {
        "id": "[OXyAsvfDQqm5f6kWJze12g][ads-202111121136][1]",
        "searches": [
          {
            "query": [
              {
                "type": "MatchAllDocsQuery",
                "description": "*:*",
                "time_in_nanos": 334356408,
                "breakdown": {
                  "set_min_competitive_score_count": 0,
                  "match_count": 0,
                  "shallow_advance_count": 0,
                  "set_min_competitive_score": 0,
                  "next_doc": 333719756,
                  "match": 0,
                  "next_doc_count": 9629720,
                  "score_count": 0,
                  "compute_max_score_count": 0,
                  "compute_max_score": 0,
                  "advance": 9127,
                  "advance_count": 25,
                  "score": 0,
                  "build_scorer_count": 50,
                  "create_weight": 8304,
                  "shallow_advance": 0,
                  "create_weight_count": 1,
                  "build_scorer": 619221
                }
              }
            ],
            "rewrite_time": 20651,
            "collector": [
              {
                "name": "PagingFieldCollector",
                "reason": "search_top_hits",
                "time_in_nanos": 531982666
              }
            ]
          }
        ],
        "aggregations": []
      }
    ]
  }
}

```

For both responses, I have intentionally removed the hits section.

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [November 16, 2021, 11:20am UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/4 "2021-11-16T11:20:03Z")

</div>

I see what's happening.

From 7.6 we [introduced an optimization](https://www.elastic.co/blog/elasticsearch-7-6-0-released) for sort queries, we rewrite a sort query as LongDistanceFeatureQuery that can skip over blocks of documents. This makes sort queries under certain conditions much faster. The optimization is currently not applied to `search_after` case. From 7.16 it will be applied to `search_after` as well, so sort queries with `search_after` also will be much faster under certain conditions.

The problem with `from/size` even though it is faster now, is extra pressure on memory when `from` number is big. When you `from=30` and `size=30` we in fact request 60 documents from every shard, and only the coordinating node after merging sorted docs from all shards can return the next 30 docs from the merged list.

---

<div class="post-metadata">

**Author:** ![manuel.pancorbo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.pancorbo/32/80928_2.png) [@manuel.pancorbo](https://discuss.elastic.co/u/manuel.pancorbo)\
**Post date:** [November 16, 2021, 11:26am UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/5 "2021-11-16T11:26:34Z")

</div>

Yeah! that makes sense since we realized that this gap between both kinds of queries is smaller as we force deeper pagination.

So, in the meanwhile, is there something that we can do to improve these queries apart from using index sorting (which fixed our issues but just for one of the sorting criteria)?

---

<div class="post-metadata">

**Author:** ![mayya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mayya/32/83147_2.png) [@mayya](https://discuss.elastic.co/u/mayya)\
**Post date:** [November 16, 2021, 2:19pm UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/6 "2021-11-16T14:19:51Z")

</div>

No, there is nothing you can do to improve the performance of queries with `search_after` except waiting for 7.16 which should be out soon.

---

<div class="post-metadata">

**Author:** ![manuel.pancorbo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/manuel.pancorbo/32/80928_2.png) [@manuel.pancorbo](https://discuss.elastic.co/u/manuel.pancorbo)\
**Post date:** [November 16, 2021, 3:08pm UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/7 "2021-11-16T15:08:42Z")

</div>

Nice, sounds good. Thanks for the help! 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 14, 2021, 3:09pm UTC](https://discuss.elastic.co/t/search-after-queries-are-not-performing-as-expected/289174/8 "2021-12-14T15:09:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.