# Which nodes handle the search threadpool **URL:** \ **Category:** Elasticsearch\ **Created:** [November 10, 2016, 8:09pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718 "2016-11-10T20:09:06Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 10, 2016, 8:09pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/1 "2016-11-10T20:09:06Z")
I'm tuning my cluster and I believe I need to increase threadpool.search.queue\_size based on [https://github.com/elastic/kibana/issues/3221](https://github.com/elastic/kibana/issues/3221), heavy query use, and our large number of shards. However, before changing this I want to make sure I understand which nodes it should be applied to. Which nodes manage this threadpool? Can I only apply this to the client nodes, or should this be applied to the data nodes? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2016, 9:30pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/2 "2016-11-10T21:30:48Z")
Increasing the threadpool is a short term fix and likely to cause other issues. If you have high shard count you should reduce that for starters. However all nodes have threadpools. ---
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 10, 2016, 9:53pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/3 "2016-11-10T21:53:45Z")
Well, I have a large number of shards because I have a large number of indexes because I have a large amount of data (close to a billion documents). I create a new index hourly, and before I did that I had other problems due to large index size degrading the cluster. So I don't think reducing the shard count is the right solution. I only have 5+1 shards per index. Based on my query load and shard count I don't think it's a short term fix, I have a large number of developers running a large number of complex visualizations against a large number of shards. I'd like to up the queue\_size to 10,000, but only on the client nodes where the incoming queries are directed. I do not want to increase the memory footprint of the data nodes as they're already heavily loaded with indexing. Can I increase the thread pool on the client nodes, or are the client nodes simply forwarding on to the data nodes requiring a change to the data nodes as well? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2016, 9:54pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/4 "2016-11-10T21:54:53Z")
> [@CVTJNII](#): > > I have a large number of shards because I have a large number of indexes because I have a large amount of data (close to a billion documents) How many shards, how many indices, how much (GB) data? ---
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 10, 2016, 10:00pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/5 "2016-11-10T22:00:17Z")
2,362 shards (5 primaries and 5 replicas per data index, plus 2 for .kibana) 237 indexes 1.32 TB of data ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2016, 10:04pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/6 "2016-11-10T22:04:20Z")
So about half a gig per shard, that's really pretty wasteful to be honest. If you reduced the count then you'd reduce the requirement to increase this threadpool by an order of magnitude. ---
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 10, 2016, 10:17pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/7 "2016-11-10T22:17:12Z")
So what would you recommend as the "sweet spot" for shard size, then? In researching it I've found a fair amount of hearsay and conflicting information. We used to run about 6-24GB/shard but that had issues with the cluster being yellow for hours if it had to move the shards for some reason. Currently my larger indexes are about 1GB/shard and, with the exception of this issue which just cropped up today, the cluster is much more stable. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 10, 2016, 10:19pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/8 "2016-11-10T22:19:09Z")
We recommend keeping them under 50GB max size. Obviously there are multiple factors though, as you have found. > [@CVTJNII](#): > > We used to run about 6-24GB/shard but that had issues with the cluster being yellow for hours if it had to move the shards for some reason. Was it causing issues? Yellow shouldn't be a problem. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 10, 2016, 10:26pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/9 "2016-11-10T22:26:48Z")
The ideal shard size will depend on the use case as well as the environment. Which version of Elasticsearch are you using? How many nodes do you have in the cluster? What type of hardware are you using? What is the specification of the nodes? ---
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 10, 2016, 10:36pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/10 "2016-11-10T22:36:52Z")
Yes, it was. While it was recovering our write rate would be reduced causing message loss as the write rate was lower than the incoming data rate. It sounds like the recommendation is the shard size we used to run and for our setup I don't wish to go back to the larger shard size, I've had fewer issues with a larger number of smaller shards. I believe this issue started today due to increasing visualization use, otherwise my ES cluster has been quite healthy and responsive with the smaller shards. What other issues would increasing the queue size cause? Based on [https://github.com/elastic/kibana/issues/3221#issuecomment-170871441](https://github.com/elastic/kibana/issues/3221#issuecomment-170871441) my current impression is the risk is in the memory footprint, which is why I was asking about which nodes will be impacted. My tribe nodes which Kibana hits are using less than 50% of their memory, so I'm comfortable increasing the footprint there. On my data nodes I'm much more concerned about memory issues. For our cluster we're running ES 2.4 with 10 data nodes, 3 masters, 3 clients, and 3 tribes. Each node has 8 cpus and 32GB of ram. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 11, 2016, 6:15am UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/11 "2016-11-11T06:15:08Z")
Having hourly indices for this volume of data seem excessive, even though the total shard count for a cluster of this size is not. If you instead e.g. used daily indices with 10 primary and 10 replica shards, you would still be able to spread out indexing across all nodes in the cluster and generate 20 shards per day instead of 240. If I calculate correctly this should give you an average shard size around 5 or 6 GB, which for most use cases I come across is fine with respect to search performance as well as recovery speed. ---
**Author:** ![CVTJNII](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cvtjnii/32/11054_2.png) [@CVTJNII](https://discuss.elastic.co/u/CVTJNII)\ **Post date:** [November 11, 2016, 1:30pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/12 "2016-11-11T13:30:58Z")
As I said I used to run daily indexes and it was not satisfactory in our environment. Going back to that config will reopen a whole host of other problems with backups and recovery from disruptions. That is not a good fix for our environment, for us it's a short term fix that will cause other problems. As such I'd really like to go back to my original question as I would like to understand the implications of tuning the queue size. I've read it can cause problems if tuned improperly but have not been able to find details about what those problems would be. I'd also like to understand which nodes hold this queue, as the answer I've gotten so far is "all nodes" which doesn't completely make sense to me with the topology of queries going to the client nodes. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 9, 2016, 1:31pm UTC](https://discuss.elastic.co/t/which-nodes-handle-the-search-threadpool/65718/13 "2016-12-09T13:31:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.