KNN spends most of its time in the rewrite. So, that is indeed the hot spot. Its where the KNN search occurs.
We do segment searches serially. So, comparing your two open KNN search tickets this is what I think is happening.
You are on a single node with a single shard. That single shard has 49 segments, each seems to be an OK size (at least a GB or so).
But, this then means, on a single node, you are exploring 49 different HNSW graphs.
In the future, we want to make KNN work in parallel on the same shard but with different segments, but right now, that doesn't happen.
I think you should try force-merging your test node to fewer segments. It doesn't have to be 1. 1 would be best, but it could take a while to complete.
So, I think:
Run it asynchronously (it will take a while), and set max_num_segments to something 10 or below. Again, 1 is best, but you would see significant improvements with 10 I think.