When I restart a node, for a simple match query the took time for the first two or three queries is much more than the following. I am trying to understand why it is that way. Is it because of some sort of caching and would it make sense to disable this caching process to run performance tests ?
It does not make sense to run performance tests if you are not using all the power of the search engine?
What do you want to measure then? What your user will see in production or the bad case?
Anyhow, yes there is some caching. Most of it is the Filesystem cache. Your OS is caching files which are read. Which is happening here behind the scene.
Also some filter caches. It's automatically managed by elasticsearch.
The idea is to measure the worst case performance. I was thinking that if the cache is specific to a query, then it would make more sense to clear the cache to get an idea of how much time it would take when there's a query which has not been encountered before.