So the diff we are seeing is between FVH and no highlighting?
We can obviously expect performance differences there so a better test might be FVH versus another choice of highlighter.
Despite its name, FVH is not always faster. IIRC it relies on pre-computed disk stores of offsets to avoid re-tokenizing document strings. This might help avoid heavy costs of analyzing lengthy texts at query time but the trade-off doesn't always work in your favour. Maybe the costs of retrieving the pre-computed offsets from disk are higher than just re-analyzing short strings for highlighting. We don't know because your benchmarking was FVH vs no highlighting.
I'd suggest trying a different choice of highlighter implementation and benchmarking FVH versus that for a more realistic comparison.