Compound sort with `missing: "_last"` silently drops/substitutes hits when every document ties on the leading field

Elastic Stack Elasticsearch
1

We found a correctness bug: sorting by two fields where the leading field is missing on every matching document returns a wrong, incomplete top-N — even though the result should be mathematically identical to sorting by the second field alone.

Sort used:

"sort": [
  { "markedTime": { "order": "desc", "missing": "_last" } },
  { "createdTime": { "order": "desc" } }
]

markedTime is missing on 100% of matching docs (confirmed via exists aggregation, doc_count: 0). So this should return the same result as sorting by createdTime alone. It doesn't — over half the true top-20 is missing, replaced by lower-ranked docs. No error, no timed_out, no shard failures.

Minimal repro:

  • 2 date-partitioned indices, 5 shards each, no replicas
  • 3000 docs/index, distinct createdTime, markedTime never set on any doc

Query A (baseline): sort: [{ "createdTime": { "order": "desc" } }]
Query B (bug): sort: [{ "markedTime": { "order": "desc", "missing": "\_last" } }, { "createdTime": { "order": "desc" } }]
Query C (workaround): same as B but "missing": "-9223372036854775808" (the literal value _last resolves to) instead of the keyword

Result: A and C are identical and correct. B diverges — only 4 of 20 hits actually belong in the true top-20.

Repeating B 3x against static data:

  • 8.15.3: same wrong result every time (deterministic)
  • 9.4.3: a different wrong result every time (non-deterministic)

Since B and C resolve to the exact same internal sort value for missing docs, this isn't about the missing value itself — it's the "_last"/"_first" keyword taking a different (buggy) code path than an explicit literal value, likely in the shard/segment skip-optimization that decides whether a shard can be skipped based on the sort field's real value range, without accounting for shards holding only missing-value documents.

Workaround: use an explicit missing value (e.g. "missing": "0") instead of "_last"/"_first". Confirmed correct on both versions tested.

Is this a known issue, or are we misusing missing:"_last" with compound sort?