# Concurrent search request to elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2015, 11:32am UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539 "2015-01-08T11:32:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![vipins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipins/32/979_2.png) [@vipins](https://discuss.elastic.co/u/vipins)\ **Post date:** [January 8, 2015, 11:32am UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/1 "2015-01-08T11:32:28Z")
What is the maximum limit on the concurrent search requests with default Elastic search server settings. I am able to perform only 5 parallel search requests in my application with default settings. how can we improve the scalability of ES server search requests apart from increasing number of node,shards and queue size in search thread pool. thanks. ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 8, 2015, 12:28pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/2 "2015-01-08T12:28:28Z")
Hello, The search threadpool size (that is, how many requests can be actually worked on at once) defaults to 3 times the number of processors. This might be reduced in future, though, see: > > > Follow-up of #9135. We initially decreased the stack size because it would end > u…p a lot of memory when there are many threads. But we also have some thread > pools that may be oversized, in particular the search thread pool. > > This commit proposes to decrease the default search thread pool size from > \`3 \* num\_procs\` to \`3 \* num\_procs / 2 + 1\`. This is large enough to be sure > that we can use all the machine resources even with a search-only work load > but not too large in order to not consume too much memory because of the stack > size and thread locals. The queue size (how many requests ES can accept before starting to reject them) defaults to 1000. From what I understand, this is per thread, so the answer to your question depends on how many processors you have and how many shards get hit by each search. For example, if a search runs on 3 indices, each with 2 shards (number of replicas is irrelevant, because the search will only hit one complete set of data) you'll get 6 requests in the threadpool per search. If you have two servers with 8 cores each, you 8_3_2=48 threads available. So the cluster can work on 8 requests at once. On top of that it can still queue round-down(2 nodes \* 1000 queue size/6 requests per search)=333 searches until it starts rejecting them. Regarding your scaling question, I can't give you a direct answer, unfortunately, because it depends on a whole lot of variables, mainly how your data, queries and hardware look like and what can be changed. The fact that your threadpool queue got full is just a symptom, it's not clear to me what happens in there. I usually see this when there are lots of indices and/or those indices have lots of shards. So a single request takes a lot of requests in the threadpool, filling it up, even if the ES cluster can keep up with the load. If that's your case increase the threadpool queue size and make sure you don't have too many shards per index. If your cluster can't keep up with the load (a monitoring tool like SPM [http://sematext.com/spm/](http://sematext.com/spm/) should show you that), then the first step is to see where is the bottleneck. Again, monitoring can give some insight: are queries too expensive, can they be optimized? do you have too many cache evictions? is the heap size too large or too small? is memory, I/O or CPU the bottleneck? Things like that. It could also be that you need more/different hardware. Finally, you can make scaling ES someone else's problem by using a hosted service like Logsene [http://sematext.com/logsene/index.html](http://sematext.com/logsene/index.html). Especially if you're using ES for log- or metric-like data, you'll get lots of features out of the box, and we expose most of the ES API to plug in your custom stuff. ## Best regards, Radu Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Thu, Jan 8, 2015 at 1:32 PM, vipins [sumitharjunan@gmail.com](mailto:sumitharjunan@gmail.com) wrote: > What is the maximum limit on the concurrent search requests with default > Elastic search server settings. > > I am able to perform only 5 parallel search requests in my application with > default settings. > > how can we improve the scalability of ES server search requests apart from > increasing number of node,shards and queue size in search thread pool. > > thanks. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702.html](http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1420716748150-4068702.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1420716748150-4068702.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_2ZoQ6bewJ\_aSxFMxAO0\_5Mdsu%3D9WA4m\_be7AfSb%3D2%2BTg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_2ZoQ6bewJ_aSxFMxAO0_5Mdsu%3D9WA4m_be7AfSb%3D2%2BTg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vipins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipins/32/979_2.png) [@vipins](https://discuss.elastic.co/u/vipins)\ **Post date:** [January 8, 2015, 1:14pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/3 "2015-01-08T13:14:47Z")
Thanks a lot for your detailed response. We have got all default settings only.Single node and 5 shards. But there are lot of indices with huge number of records. search settings: "threads" : 12, "queuesize" : 1000, My query is very simple. which runs on a single index only. Even with 5 requests in between it is throwing None of the configured nodes are available. Thanks, ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 8, 2015, 1:30pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/4 "2015-01-08T13:30:53Z")
You're welcome. So you're saying you're running 5 searches on a single index with 5 shards (25 per-shard queries in total) and you're getting an error? I assume that error doesn't say the queue is full because the queue is 1000. Can you post the full error and also a gist where you reproduce the issue [http://www.elasticsearch.org/help](http://www.elasticsearch.org/help)? I might be missing an essential bit here. ## Best regards, Radu Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Thu, Jan 8, 2015 at 3:14 PM, vipins [sumitharjunan@gmail.com](mailto:sumitharjunan@gmail.com) wrote: > Thanks a lot for your detailed response. > We have got all default settings only.Single node and 5 shards. But there > are lot of indices with huge number of records. > search settings: > "threads" : 12, > "queuesize" : 1000, > > My query is very simple. which runs on a single index only. > > Even with 5 requests in between it is throwing None of the configured nodes > are available. > > Thanks, > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068707.html](http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068707.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1420722888084-4068707.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1420722888084-4068707.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_0DMNN91Xo\_iQpFFdep%3DosynTs5tGr-UDX\_EOCOX%3DTg5Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_0DMNN91Xo_iQpFFdep%3DosynTs5tGr-UDX_EOCOX%3DTg5Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vipins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipins/32/979_2.png) [@vipins](https://discuss.elastic.co/u/vipins)\ **Post date:** [January 8, 2015, 1:46pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/5 "2015-01-08T13:46:19Z")
Sorry , I was wrong with number of shards. actual number of shards is 320 for the index which i am querying. We are using rolling indices on a daily basis. max queue size is 1000 for search thread pool. We overcome the issue None of the configured nodes are available by keeping tcp connection alive as true. ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 8, 2015, 1:58pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/6 "2015-01-08T13:58:45Z")
OK, now it makes sense. 5 requests with 320 shards might saturate your queue. But 320 shards sounds like a lot for one index. I assume you don't need to scale that very index to 320 nodes (+ replicas). If you can get the number of shards down (say, to the default of 5) things will surely look better not only from the queue's perspective, but it should also improve search performance. -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Thu, Jan 8, 2015 at 3:46 PM, vipins [sumitharjunan@gmail.com](mailto:sumitharjunan@gmail.com) wrote: > Sorry , I was wrong with number of shards. actual number of shards is 320 > for > the index which i am querying. > > We are using rolling indices on a daily basis. > > max queue size is 1000 for search thread pool. > > We overcome the issue None of the configured nodes are available by keeping > tcp connection alive as true. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068711.html](http://elasticsearch-users.115913.n3.nabble.com/concurrent-search-request-to-elasticsearch-tp4068702p4068711.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/1420724779413-4068711.post%40n3.nabble.com](https://groups.google.com/d/msgid/elasticsearch/1420724779413-4068711.post%40n3.nabble.com) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_00wG%2BNUQQm2\_KtH7jKBC7ovN1AXnAf9Jot2VCTppMk9g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_00wG%2BNUQQm2_KtH7jKBC7ovN1AXnAf9Jot2VCTppMk9g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vipins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipins/32/979_2.png) [@vipins](https://discuss.elastic.co/u/vipins)\ **Post date:** [January 8, 2015, 2:01pm UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/7 "2015-01-08T14:01:47Z")
Thanks for your prompt response. Surely will reduce the number of shards with nodes/replicas addition for the better performance of the search. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:40am UTC](https://discuss.elastic.co/t/concurrent-search-request-to-elasticsearch/21539/8 "2017-07-06T00:40:02Z")