I am debugging a long query and encountered below observation.
The actual query takes 4.976 seconds.
But the kibana search profiler states the search takes only 23.416 ms and aggregation takes 574.025 ms cumulatively each.
I copy & pasted the payload so I'm sure the query strings are identical. And caching is not the issue here.
In theory the search time should be shorter than the sum of the 2 cumulative values since I have more nodes than shard counts. So parallel processing is in place.
What do I make of this? Any suggestions on how to dig deeper? Thanks.
*note: The profiler also takes multiple seconds to come back with the result. Not sure if it makes any difference here.