We are currently using the unified highlighter but have noticed some edge cases whereby the unified highlighter takes too long to return results in extremely large documents with lots of matches for a specific search term. This is presumably a consequence of using "order: score" as there are so many matches that it takes prohibitively long to sort the resulting fragments.
As one remediation, we have experimented with using fast vector highlighting. I was expecting search performance in general to improve, however I measured a slight regression in performance for most cases. The avg response time for queries increased from roughly 10ms to 16ms:
The median also increased but the p99 decreased somewhat.
Can you explain what factors are at work here?