Search_after returning duplicates and limited results

[carlos_funk (Carl Robben)]

I'm having issues with search_after functionality returning duplicate results and also not returning the full set of results.

I've followed the documentation and am using a unique id (url_id) for each document, plus setting a preference to a uuid4 value for each user but the results coming back often have duplicates. I've also noticed that even though the total number of results might report for example 100 it starts returning 0 results after only scrolling through 80.

I only see this in my production environment with 3 shards and one replica for each index, in dev and locally with 1 shard and 0 replicas I can't reproduce it.

Here's a somewhat simplified example showing two requests, the second request returned docs that were also returned in the first:

GET http://localhost:9200/item/doc/_search?preference=9621e3124b0044728a7833dc03512f89

{
    "query": {
        "bool": {
            "must": [
                {"match": {"niche_slug": "collectables"}},
                {"match": {"section_slug": "competitions-gaming"}},
                {"match": {"gallery_slug": "guess-the-price"}},
            ],
            "filter": [
                {"terms": {"groups": ["Public"]}},
                {"term": {"searchable": "true"}},
            ],
        }
    },
    "sort": [{"created_date": {"order": "desc"}}, "url_id"],
    "from": 0,
    "size": 11,
    "search_after": ["1555022693471", "1365999"]
}

GET http://localhost:9200/item/doc/_search?preference=9621e3124b0044728a7833dc03512f89

{
    "query": {
        "bool": {
            "must": [
                {"match": {"niche_slug": "collectables"}},
                {"match": {"section_slug": "competitions-gaming"}},
                {"match": {"gallery_slug": "guess-the-price"}},
            ],
            "filter": [
                {"terms": {"groups": ["Public"]}},
                {"term": {"searchable": "true"}},
            ],
        }
    },
    "sort": [{"created_date": {"order": "desc"}}, "url_id"],
    "from": 0,
    "size": 11,
    "search_after": ["1540412268500", "1320805"],
}

Any help would be appreciated.