The query phase visits each segment independently but terminate_after is a global count per shard so there's no guarantee that you'll see a sorted stream even if the index is sorted. As you already noticed it works if you force merge all shards to 1 segment but that's just a side-effect of the fact that the index is sorted, not a bug since terminate_after does not change the way documents are collected.
Today the only way to achieve what you want would be to retrieve an extended top N sorted by the index sort criteria and to sort/prune this result client-side using the value of a field extracted from the _source or from the doc values. This is not ideal so I think it's worth opening a feature request in github to discuss a full solution that would not require extra logic in the client. Could you open one ?