Elasticsearch Sorting before applying filters?

Hello,

I have noticed that when I'm applying sorting in my queries, the return times are remaining relatively constant at ~1.5s no matter what filters I apply (Even if I get 0 hits). It seems like ES might be sorting the entire index, then applying the filters after.

As a test I checked how long it took ES to sort the entire index and it was about 2s.

Here are some results for different queries comparing how many hits they got to how much time they took(ms):

Took: 1563
Hits: 7000

Took: 1660
Hits: 239360

Took: 1657
Hits: 0

As you can see the time the queries are taking remains relatively constant when I have sorting on no matter how many hits I get. What is happening here?

As a baseline if I remove sorting I get the following:

Took: 150
Hits: 7000

Took: 134
Hits: 239360

Took: 0
Hits: 0

*All these tests were done with a return size of 500, so I would expect queries with more hits to return slightly faster