I have created a custom rescorer, following the examples of QueryRescorer / ExampleRescorer.
It works fine, but the latency added to the responses is much bigger than it seemingly should be.
I am using a test query, which consistently takes around 40ms to execute (without rescoring).
When I add my custom rescorer, the response times shoot up to ~130ms.
I have profiled the rescore() method of my custom class and it is consistently executed within 30-35ms. I would expect the total query time to be close to the sum of the plain query and the rescoring, ie. 70-75ms.
What could possibly induce the extra overhead? I looked through the rescoring-related code, but could not find anything suspicious.
Any help would be greatly appreciated.