# Query time increasing with increasing number of replicas and servers **URL:** \ **Category:** Elasticsearch\ **Created:** [October 18, 2012, 3:49pm UTC](https://discuss.elastic.co/t/query-time-increasing-with-increasing-number-of-replicas-and-servers/9410 "2012-10-18T15:49:41Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Zmicier](https://avatars.discourse-cdn.com/v4/letter/z/ecc23a/32.png) [@Zmicier](https://discuss.elastic.co/u/Zmicier)\ **Post date:** [October 18, 2012, 3:49pm UTC](https://discuss.elastic.co/t/query-time-increasing-with-increasing-number-of-replicas-and-servers/9410/1 "2012-10-18T15:49:41Z")
I am seeing unexpected elastic behavior: with increasing the number of replicas _and_ servers, the query time is actually increasing, while I would expect it to decrease. I have a cluster with 8 million complex documents (overall about 300 million nested documents). I run complex queries, where each query involves computing dozens of terms facets (top 100 values) on non-analyzed fields with hundreds of thousands unique field values. I have a controlled load testing environment, where I fire a number of queries simultaneously and average measure query time. Since queries did not run very fast (5-10 seconds/query on average), I started adding servers and replicas. However, not only did this not help query time, but it made things worse, as you can see from the attached graph. Any ideas of why this is happening? -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [October 18, 2012, 8:05pm UTC](https://discuss.elastic.co/t/query-time-increasing-with-increasing-number-of-replicas-and-servers/9410/2 "2012-10-18T20:05:38Z")
Hello Zmicier, Maybe the OS cache or Elasticsearch's own caches play a role here. How many queries did you run for each test to get the average? Another thing might be the distribution of load. Do you get different results if you add a non-data node as a load balancer and run the same queries on it? ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Thu, Oct 18, 2012 at 6:49 PM, Zmicier [zialonka@gmail.com](mailto:zialonka@gmail.com) wrote: > I am seeing unexpected elastic behavior: with increasing the number of > replicas and servers, the query time is actually increasing, while I would > expect it to decrease. > > I have a cluster with 8 million complex documents (overall about 300 million > nested documents). I run complex queries, where each query involves > computing dozens of terms facets (top 100 values) on non-analyzed fields > with hundreds of thousands unique field values. I have a controlled load > testing environment, where I fire a number of queries simultaneously and > average measure query time. Since queries did not run very fast (5-10 > seconds/query on average), I started adding servers and replicas. However, > not only did this not help query time, but it made things worse, as you can > see from the attached graph. > > Any ideas of why this is happening? > > -- -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [October 19, 2012, 2:02am UTC](https://discuss.elastic.co/t/query-time-increasing-with-increasing-number-of-replicas-and-servers/9410/3 "2012-10-19T02:02:25Z")
Hi Zmicier, Bizarre indeed. Have a look at some other metrics, both ES and system one, like disk IO, memory, CPU, etc. How do they change as you add more servers? Did you check if shards were evenly spread in the cluster and there were no shards being moved around the cluster when you ran your tests? If you don't have a tool that can show you that, [Elasticsearch - Sematext Documentation](http://sematext.com/spm/elasticsearch-performance-monitoring/index.html) will help. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, October 18, 2012 11:49:41 AM UTC-4, Zmicier wrote: > I am seeing unexpected elastic behavior: with increasing the number of > replicas _and_ servers, the query time is actually increasing, while I > would expect it to decrease. > > I have a cluster with 8 million complex documents (overall about 300 > million nested documents). I run complex queries, where each query involves > computing dozens of terms facets (top 100 values) on non-analyzed fields > with hundreds of thousands unique field values. I have a controlled load > testing environment, where I fire a number of queries simultaneously and > average measure query time. Since queries did not run very fast (5-10 > seconds/query on average), I started adding servers and replicas. However, > not only did this not help query time, but it made things worse, as you can > see from the attached graph. > > Any ideas of why this is happening? -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:07am UTC](https://discuss.elastic.co/t/query-time-increasing-with-increasing-number-of-replicas-and-servers/9410/4 "2017-07-06T03:07:57Z")