I am seeing unexpected elastic behavior: with increasing the number of
replicas and servers, the query time is actually increasing, while I
would expect it to decrease.
I have a cluster with 8 million complex documents (overall about 300
million nested documents). I run complex queries, where each query involves
computing dozens of terms facets (top 100 values) on non-analyzed fields
with hundreds of thousands unique field values. I have a controlled load
testing environment, where I fire a number of queries simultaneously and
average measure query time. Since queries did not run very fast (5-10
seconds/query on average), I started adding servers and replicas. However,
not only did this not help query time, but it made things worse, as you can
see from the attached graph.
Any ideas of why this is happening?