Update By Query - status counts do not add up to total

I'm running ES 7.17.3 with an index with 2.1M documents. I executed an update-by-query request with a script to clean up some data (there was no query, so it ran against all documents in the index). The script sets ctx.op to "noop" if it finds nothing to update but otherwise does not set this field (I understand that it defaults to update).

As far as I can tell, my request was successful with the exception of 15 conflicts (which is expected due to continuous data loading) but I expected the "total" to be equal to the sum of the other categories ("updated", "created", "deleted", "noop", and "version_conflicts"). But, there are nearly 100k documents counted in the total that don't appear to be counted in the categories. Is this expected?

Here's the abridged output from the completed task status. Should I expect these counts to match one another?

{
    "completed": true,
    "task": {
        "node": "GL5lCqkeRrGuWq4bBZYidA",
        "id": 18025434,
        "type": "transport",
        "action": "indices:data/write/update/byquery",
        "status": {
            "total": 2111746,
            "updated": 218896,
            "created": 0,
            "deleted": 0,
            "batches": 402722,
            "version_conflicts": 15,
            "noops": 1794693,
            "retries": {
                "bulk": 0,
                "search": 0
            },
            "throttled_millis": 0,
            "requests_per_second": -1,
            "throttled_until_millis": 0,
            "slices": []
        },
        "description": "update-by-query [index] updated with Script{type=stored, lang='null', idOrCode='expired-cleanup', options=null, params={now=2022-10-03T14:57:30Z, scriptName=expired-cleanup}}",
        "start_time_in_millis": 1664809050759,
        "running_time_in_nanos": 4920791773250,
        "cancellable": true,
        "cancelled": false,
        "headers": {}
    },
    "response": {
        "took": 4920779,
        "timed_out": false,
        "total": 2111746,
        "updated": 218896,
        "created": 0,
        "deleted": 0,
        "batches": 402722,
        "version_conflicts": 15,
        "noops": 1794693,
        "retries": {
            "bulk": 0,
            "search": 0
        },
        "throttled": "0s",
        "throttled_millis": 0,
        "requests_per_second": -1,
        "throttled_until": "0s",
        "throttled_until_millis": 0,
        "slices": [],
        "failures": []
    }
}

So, total reflects the # of documents in my index (2111746) but the various categories only add up to 2013504.

The documentation for update-by-query states that these totals should match, at least in the context of the example being explained there, but maybe that's not the whole story.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/docs-update-by-query.html#CO611-1

Thanks in advance for any clarification.

Bob

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.