Hmm, a proportionate slowdown isn't what I expected, but perhaps there's more than one effect here.
The main contributor will be, I think, the interaction of index.refresh_interval and index.search.idle.after. If you do not search a shard for 30 seconds (by default) then it will stop refreshing to save resources. The next search that you do will trigger a refresh within the next second (by default) and will wait for that to complete first.
Perhaps the best way to test whether this is the problem is to increase index.search.idle.after to a longer time, say 10 minutes, and see whether this speeds things up for searches spaced 30-60 seconds apart. Could you try that and see if it helps?