Search performance difference between from&size and search&after

Elastic Stack Elasticsearch
1

There are hundreds of millions of documents in my index. When I search, I find that search&after is much slower than from&size。 Use from&size，search is quick，took several ms return，but use search after，it took 20 seconds。Search is sort by time and key(a copy of _id) ，Why？what's the difference?
search cmd:

{
    "query":{
        "bool":{
            "filter":[
                {
                    "query_string":{
                        "query":"*"
                    }
                },
                {
                    "range":{
                        "__time__":{
                            "gte":1324958207,
                            "lte":1724958207
                        }
                    }
                }
            ]
        }
    },
    "size":10,
    "sort":[
        {
            "__time__":{
                "order":"desc"
            }
        },
        {
            "__key__":{
                "order":"desc"
            }
        }
    ],
    "search_after":[
        1630594662000,
        "6130e666-2-67e9e3-f5-1"
    ],
    "profile":true
}

profile:

 "searches":[
                    {
                        "query":[
                            {
                                "type":"BoostQuery",
                                "description":"(ConstantScore(DocValuesFieldExistsQuery [field=__time__]))^0.0",
                                "time_in_nanos":45722536283,
                                "breakdown":{
                                    "set_min_competitive_score_count":0,
                                    "match_count":0,
                                    "shallow_advance_count":0,
                                    "set_min_competitive_score":0,
                                    "next_doc":45722241414,
                                    "match":0,
                                    "next_doc_count":410919487,
                                    "score_count":0,
                                    "compute_max_score_count":0,
                                    "compute_max_score":0,
                                    "advance":19517,
                                    "advance_count":39,
                                    "score":0,
                                    "build_scorer_count":78,
                                    "create_weight":14271,
                                    "shallow_advance":0,
                                    "create_weight_count":1,
                                    "build_scorer":261081
                                },
                                "children":[
                                    {
                                        "type":"DocValuesFieldExistsQuery",
                                        "description":"DocValuesFieldExistsQuery [field=__time__]",
                                        "time_in_nanos":16571715415,
                                        "breakdown":{
                                            "set_min_competitive_score_count":0,
                                            "match_count":0,
                                            "shallow_advance_count":0,
                                            "set_min_competitive_score":0,
                                            "next_doc":16571493898,
                                            "match":0,
                                            "next_doc_count":410919487,
                                            "score_count":0,
                                            "compute_max_score_count":0,
                                            "compute_max_score":0,
                                            "advance":15074,
                                            "advance_count":39,
                                            "score":0,
                                            "build_scorer_count":78,
                                            "create_weight":517,
                                            "shallow_advance":0,
                                            "create_weight_count":1,
                                            "build_scorer":205926
                                        }
                                    }
                                ]
                            }
                        ],
                        "rewrite_time":116538,
                        "collector":[
                            {
                                "name":"PagingFieldCollector",
                                "reason":"search_top_hits",
                                "time_in_nanos":30851166561
                            }
                        ]
                    }
                ],
                "aggregations":[

                ]
            }
2

just to be sure, have you compared both searches after a warmup period and both searches when compared either include or both do not include the profile boolean set?

Also, have you compared the profile output of both (there is a tool in kibana dev-tools that makes the output a little more graphical and easier to read).

In general using search after should require much less memory when paginated deeper (on the coordinating node), but both operations have to execute the same search.

I also guess the query_string in there is just an example, as otherwise you could go with a match_all query or none as well?

Also, which Elasticsearch version are you on?

3

Thanks for reply，Elasticsearch version is 7.10.

Query string is just an example, match_all same result .

4

can you please answer the other questions above as well to get a better picture? Thank you!

1 Like
5

image
image2966×292 103 KB

left result is without search-after，right result is with search after。
query type is different， BooleanQuery vs boostQuery。
description is different，left is DocValuesFieldExistsQuery&LongDistanceFeatureQuery，right is just DocValuesFieldExistsQuery。
image
image3402×1080 498 KB

children query also has some difference。

6

full left profile, with out search-after

--------------------
 "searches":[
                    {
                        "query":[
                            {
                                "type":"BooleanQuery",
                                "description":"#(ConstantScore(DocValuesFieldExistsQuery [field=__time__]))^0.0 LongDistanceFeatureQuery(field=,origin=1630594662000,pivotDistance=43183000)",
                                "time_in_nanos":50216920945,
                                "breakdown":{
                                    "set_min_competitive_score_count":5076,
                                    "match_count":0,
                                    "shallow_advance_count":0,
                                    "set_min_competitive_score":615806781,
                                    "next_doc":20485284262,
                                    "match":0,
                                    "next_doc_count":193283578,
                                    "score_count":193283578,
                                    "compute_max_score_count":0,
                                    "compute_max_score":0,
                                    "advance":60903,
                                    "advance_count":39,
                                    "score":29114449991,
                                    "build_scorer_count":78,
                                    "create_weight":52360,
                                    "shallow_advance":0,
                                    "create_weight_count":1,
                                    "build_scorer":1266648
                                },
                                "children":[
                                    {
                                        "type":"BoostQuery",
                                        "description":"(ConstantScore(DocValuesFieldExistsQuery [field=__time__]))^0.0",
                                        "time_in_nanos":19146981415,
                                        "breakdown":{
                                            "set_min_competitive_score_count":0,
                                            "match_count":0,
                                            "shallow_advance_count":0,
                                            "set_min_competitive_score":0,
                                            "next_doc":0,
                                            "match":0,
                                            "next_doc_count":0,
                                            "score_count":0,
                                            "compute_max_score_count":0,
                                            "compute_max_score":0,
                                            "advance":19146741485,
                                            "advance_count":193511946,
                                            "score":0,
                                            "build_scorer_count":117,
                                            "create_weight":25754,
                                            "shallow_advance":0,
                                            "create_weight_count":1,
                                            "build_scorer":214176
                                        },
                                        "children":[
                                            {
                                                "type":"DocValuesFieldExistsQuery",
                                                "description":"DocValuesFieldExistsQuery [field=__time__]",
                                                "time_in_nanos":6894905065,
                                                "breakdown":{
                                                    "set_min_competitive_score_count":0,
                                                    "match_count":0,
                                                    "shallow_advance_count":0,
                                                    "set_min_competitive_score":0,
                                                    "next_doc":0,
                                                    "match":0,
                                                    "next_doc_count":0,
                                                    "score_count":0,
                                                    "compute_max_score_count":0,
                                                    "compute_max_score":0,
                                                    "advance":6894734027,
                                                    "advance_count":193511946,
                                                    "score":0,
                                                    "build_scorer_count":117,
                                                    "create_weight":625,
                                                    "shallow_advance":0,
                                                    "create_weight_count":1,
                                                    "build_scorer":170413
                                                }
                                            }
                                        ]
                                    },
                                    {
                                        "type":"LongDistanceFeatureQuery",
                                        "description":"LongDistanceFeatureQuery(field=,origin=1630594662000,pivotDistance=43183000)",
                                        "time_in_nanos":19735814288,
                                        "breakdown":{
                                            "set_min_competitive_score_count":5076,
                                            "match_count":0,
                                            "shallow_advance_count":117,
                                            "set_min_competitive_score":614826804,
                                            "next_doc":0,
                                            "match":0,
                                            "next_doc_count":0,
                                            "score_count":193283578,
                                            "compute_max_score_count":78,
                                            "compute_max_score":4210,
                                            "advance":7497803263,
                                            "advance_count":193283611,
                                            "score":11622998714,
                                            "build_scorer_count":78,
                                            "create_weight":7033,
                                            "shallow_advance":6887,
                                            "create_weight_count":1,
                                            "build_scorer":167377
                                        }
                                    }
                                ]
                            }
                        ],
                        "rewrite_time":290412,
                        "collector":[
                            {
                                "name":"SimpleFieldCollector",
                                "reason":"search_top_hits",
                                "time_in_nanos":30033925274
                            }
                        ]
                    }
                ],
                "aggregations":[

                ]
            }
        ]

Full right profile with search-after


                "searches":[
                    {
                        "query":[
                            {
                                "type":"BoostQuery",
                                "description":"(ConstantScore(DocValuesFieldExistsQuery [field=__time__]))^0.0",
                                "time_in_nanos":52342571431,
                                "breakdown":{
                                    "set_min_competitive_score_count":0,
                                    "match_count":0,
                                    "shallow_advance_count":0,
                                    "set_min_competitive_score":0,
                                    "next_doc":52342129926,
                                    "match":0,
                                    "next_doc_count":410870383,
                                    "score_count":0,
                                    "compute_max_score_count":0,
                                    "compute_max_score":0,
                                    "advance":30398,
                                    "advance_count":39,
                                    "score":0,
                                    "build_scorer_count":78,
                                    "create_weight":26799,
                                    "shallow_advance":0,
                                    "create_weight_count":1,
                                    "build_scorer":384308
                                },
                                "children":[
                                    {
                                        "type":"DocValuesFieldExistsQuery",
                                        "description":"DocValuesFieldExistsQuery [field=__time__]",
                                        "time_in_nanos":20595411270,
                                        "breakdown":{
                                            "set_min_competitive_score_count":0,
                                            "match_count":0,
                                            "shallow_advance_count":0,
                                            "set_min_competitive_score":0,
                                            "next_doc":20595073961,
                                            "match":0,
                                            "next_doc_count":410870383,
                                            "score_count":0,
                                            "compute_max_score_count":0,
                                            "compute_max_score":0,
                                            "advance":20634,
                                            "advance_count":39,
                                            "score":0,
                                            "build_scorer_count":78,
                                            "create_weight":7160,
                                            "shallow_advance":0,
                                            "create_weight_count":1,
                                            "build_scorer":309515
                                        }
                                    }
                                ]
                            }
                        ],
                        "rewrite_time":220755,
                        "collector":[
                            {
                                "name":"PagingFieldCollector",
                                "reason":"search_top_hits",
                                "time_in_nanos":36757066196
                            }
                        ]
                    }
                ],
                "aggregations":[

                ]
            }
        ]
7

Thanks very much!

just to be sure, have you compared both searches after a warmup period and both searches when compared either include or both do not include the profile boolean set?
-------- I have tested more than 10 times, and the results are the same every time, which has nothing to do with warm up。

Also, have you compared the profile output of both (there is a tool in kibana dev-tools that makes the output a little more graphical and easier to read).
-------see the above reply

In general using search after should require much less memory when paginated deeper (on the coordinating node), but both operations have to execute the same search.
------both are the same search，with or without search_after，search_after context offset is 10.

{
    "query":{
        "bool":{
            "must":{
                "match_all":{

                }
            },
            "filter":[
                {
                    "range":{
                        "__time__":{
                            "gte":1324958207,
                            "lte":1724958207
                        }
                    }
                }
            ]
        }
    },
    "size":10,
    "sort":[
        {
            "__time__":{
                "order":"desc"
            }
        },
        {
            "__key__":{
                "order":"desc"
            }
        }
    ]
//,
//   "search_after":[
//      1630594662000,
//     "6130e666-2-67e9e3-f5-1"
// ]
}

I also guess the query_string in there is just an example, as otherwise you could go with a match_all query or none as well?
-----I have tried match_all query, same result.

Also, which Elasticsearch version are you on? ---7.10

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.