Fascinating! Thank you for the hint. I never would have expected that. I will have to run some more tests. I assumed that because the index had so few docs that it wasn't worth even thinking about the number of primary shards and assumed that replicas would do most of the work of helping spread the search load (just like in "regular" queries).
It is really interesting all the different ways people use these features. As you stated, our use-cases are quite literally the opposite. We index hundreds of millions of docs against a relatively small number of queries.
During our first benchmarks with the ES 5.X releases we had truly abysmal performance in the low hundreds of DPS. So instead we wrote application logic to regularly search the most recently indexed documents against our saved queries (sort of a manual, batch, percolate system). The performance was reasonable, but was not something we wanted to continue to maintain and performance started to decrease as well.
Thank you. I remember reading that and believe that we wrap all of our queries in a bool filter, but I'll double check.