# Change thread pool search queue\_size? yes or not? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 16, 2017, 2:56pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284 "2017-08-16T14:56:42Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [August 16, 2017, 2:56pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/1 "2017-08-16T14:56:43Z")
Got one question. Working on my elastic stack. Basically it's "developing production". Got one server with 16GB of RAM, 4 CPUs, ELK 5.4.0. No cluster or extra nodes. Got no problems with adding data and searching, the problems gets with dashboards with multi visualizations. I figure, the problem is probably CPU? With development, also the document size increased as I try to automate as much as possible. So indexes are created daily with around 5-6 mio documents per index. So with few months of data I'm already at several 100 millions of documents. So when I open a dashboard with multiple visualisation I receive a timeout error in kibana and elasticsearch log says: `Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.transport.TransportService$7@59b0d1f1 on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@395e3b7a[Running, pool size = 7, active threads = 7, queued tasks = 1000, completed tasks = 3105230]]` So i'm thinking of maybe increase the search queue\_size. Would that even make sense? What would be the optimal size in my situation? Best would be probably to add two more nodes and create a cluster. Thanks! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2017, 3:06pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/2 "2017-08-16T15:06:32Z")
How many indices and shards are you creating per day? What is your average shard size? ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [August 16, 2017, 3:11pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/3 "2017-08-16T15:11:01Z")
1 index and 1 shard per day. Size of data in index is around 2 GB per index. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2017, 3:18pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/4 "2017-08-16T15:18:48Z")
That sounds quite reasonable. How many visualisations do you have on the affected dashboards? Do you have X-Pack Monitoring installed? What does CPU usage and disk I/O look like when you experience the timeout? ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [August 16, 2017, 6:24pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/5 "2017-08-16T18:24:18Z")
5 visualization. Yes, x-pack is installed. The only thing I really see out of the ordinary is the System load which rose over 8.00. Then the elasticsearch just quit/killed for few minutes. So my asumption is lackage of CPU? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 16, 2017, 6:25pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/6 "2017-08-16T18:25:32Z")
What is your heap size? What does GC graph look like in monitoring? ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [August 17, 2017, 7:01am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/7 "2017-08-17T07:01:40Z")
-Xms8g -Xmx8g GC count young jumped from 8 to 23, duration from 230 ms to 781 ms, Cgroup CPU utilization from 26% to 66% and Cgroup usage from 35b ns to 53.7b ns. Other graphs had no significant changes. ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [September 5, 2017, 2:17pm UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/8 "2017-09-05T14:17:02Z")
You have any tips or suggestions? tnx ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2017, 6:44am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/9 "2017-09-06T06:44:35Z")
How large are your indices/shards? How many are you querying across when you encounter problems? ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [September 6, 2017, 7:07am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/10 "2017-09-06T07:07:47Z")
Indices are on average around 2,5 GB with 6mio documents, created daily. Default search/overview is last 24 hours. Increased the cpu cores, but still encounter time outs. Even with only 2-3 visualizations on dashboard. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2017, 7:12am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/11 "2017-09-06T07:12:42Z")
What type of storage do you have? What does disk I/O and iowait look like? How many concurrent queries? ---
**Author:** ![mitabrev](https://avatars.discourse-cdn.com/v4/letter/m/a88e57/32.png) [@mitabrev](https://discuss.elastic.co/u/mitabrev)\ **Post date:** [September 6, 2017, 8:05am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/12 "2017-09-06T08:05:13Z")
Looks like a disk reading issue... As soon as I open dashboard the values increase... From: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r\_await w\_await svctm %util sda 0.00 0.00 0.00 0.80 0.00 8.00 20.00 0.00 0.00 0.00 0.00 0.00 0.00 sdb 0.00 0.00 0.00 2.20 0.00 16.20 14.73 0.00 0.55 0.00 0.55 0.36 0.08 To: Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r\_await w\_await svctm %util sda 0.00 659.20 36.80 25.40 946.40 2744.00 118.66 0.29 4.73 4.66 4.83 1.54 9.60 sdb 0.00 0.40 3645.00 11.00 1582319.20 46.70 865.63 130.93 35.73 35.77 20.13 0.27 99.98 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 6, 2017, 8:20am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/13 "2017-09-06T08:20:52Z")
Yes, it looks like your storage is indeed the bottleneck. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2017, 8:21am UTC](https://discuss.elastic.co/t/change-thread-pool-search-queue-size-yes-or-not/97284/14 "2017-10-04T08:21:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.