Are you using es 7.12 version where
_shard_doc was introduced?
The documentation you linked has information about this field:
_shard_doc value is the combination of the shard index within the PIT and the Lucene’s internal doc ID, it is unique per document and constant within a PIT.
This is an automatic tie-breaker field, and is composed of Lucene internal doc ID and a shard ID. It makes sense only for searches with PIT, as PIT represents a point of time of the index, where Lucene internal doc IDs stay constant. Thus, if you use the same PIT request multiple times, you will always get the same order of documents.
You don't need to explicitly add
sort with PIT search, it will be automatically added as
"_shard_doc": "asc". The only way, where you want to explicitly add it, if you paginate backwards (going to previous results), then as an example shows you would add
In the past before PIT was introduced, as index was changing, the same sort request may produce different results. Also with
search_after we could get repeated results from previous page.
Another problem was how to tie-breaking equal values. In the past, we were using expensive ways – unique fields such as elasticsearch doc
_id, which represented long strings that need to be loaded into memory and compared.
_shard_doc field is a cheap way to do tie-breaking.