Rescoring adds unexplained latency to queries

I have created a custom rescorer, following the examples of QueryRescorer / ExampleRescorer.

It works fine, but the latency added to the responses is much bigger than it seemingly should be.

I am using a test query, which consistently takes around 40ms to execute (without rescoring).

When I add my custom rescorer, the response times shoot up to ~130ms.

I have profiled the rescore() method of my custom class and it is consistently executed within 30-35ms. I would expect the total query time to be close to the sum of the plain query and the rescoring, ie. 70-75ms.

What could possibly induce the extra overhead? I looked through the rescoring-related code, but could not find anything suspicious.

Any help would be greatly appreciated.