# Cluster has pending tasks. Cannot update cluster settings **URL:** \ **Category:** Elasticsearch\ **Created:** [December 4, 2013, 3:51pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706 "2013-12-04T15:51:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jenny\_Sivapalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenny_sivapalan/32/1884_2.png) [@Jenny\_Sivapalan](https://discuss.elastic.co/u/Jenny_Sivapalan)\ **Post date:** [December 4, 2013, 3:51pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/1 "2013-12-04T15:51:24Z")
Hello, We've managed to get our cluster into a state where it has pending tasks and we unable to update cluster settings. Has anyone else experienced this and be able to help us to remove these tasks? We're thinking the alternative option is to kill the cluster and start again but we'd like to avoid this. We got to this state by making changes to the threadpool search queue size. It's default is 1000 to unbounded (-1) which applied without issue. We ran a load test on our application. Next we changed the threadpool search queue size to 500 but this looks like it did not complete although we can see it has been applied in the :9200/\_cluster/setting. The curl command we used to apply the thread pool settings was: curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { "threadpool.search.queue\_size" : 500}}' Logs for updating the threadpool search queue size from 1000 to -1: 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] creating thread\_pool [search], type [fixed], size [40], queue\_size [-1] 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] processing [zen-disco-receive(from master [[Cap 'N Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying updated cluster\_state (version: 1598) Logs for updating the threadpool search queue from -1 to 500 (Note it never applied.): 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] creating thread\_pool [search], type [fixed], size [40], queue\_size [500] When we've tried to update the threadpool search queue size again we get see it being added to the pending tasks list but is then rejected. The error we get is:- ProcessClusterEventTimeoutException[failed to process cluster event (cluster\_update\_settings) within 30s]; ","status":503} Looking at the pending tasks (:9200/\_cluster/pending\_tasks) shows (the cluster\_update\_settings goes in with a higher insert order number than the ones listed below):- { "tasks": [ { "insert\_order": 33, "priority": "HIGH", "source": "update-mapping [][]", "time\_in\_queue\_millis": 82081320, "time\_in\_queue": "22.8h" }, { "insert\_order": 40, "priority": "HIGH", "source": "update-mapping [][]", "time\_in\_queue\_millis": 64831934, "time\_in\_queue": "18h" } ] } We are running Elastic Search 0.90.7. Thanks, Jenny -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7d2767ed-1256-470c-9825-f34e63a28a2c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d2767ed-1256-470c-9825-f34e63a28a2c%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jenny\_Sivapalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenny_sivapalan/32/1884_2.png) [@Jenny\_Sivapalan](https://discuss.elastic.co/u/Jenny_Sivapalan)\ **Post date:** [December 5, 2013, 2:36pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/2 "2013-12-05T14:36:06Z")
We ended up following the workaround listed on github to resolve this:- > > > There's no delete API for these, if you want to set them back to defaults on nex…t startup. > > karmi on irc suggested setting them to the empty string, e.g.: > > \`\`\` > curl -XPUT 'localhost:9200/\_cluster/settings' -d '{"persistent":{"threadpool.search.type":""}}' > \`\`\` > > But then on next restart I got warnings like this: > > \`\`\` > \[2013-09-11 20:08:41,548\]\[WARN \]\[node.settings \] \[lo3uppaldbs009.palomino.pearson.com\] failed to refresh settings for \[org.elasticsearch.threadpool.ThreadPool$ApplySettings@49e3f731\] > org.elasticsearch.ElasticSearchIllegalArgumentException: No type found \[\], for \[search\] > at org.elasticsearch.threadpool.ThreadPool.rebuild(ThreadPool.java:452) > at org.elasticsearch.threadpool.ThreadPool.updateSettings(ThreadPool.java:468) > at org.elasticsearch.threadpool.ThreadPool$ApplySettings.onRefreshSettings(ThreadPool.java:809) > at org.elasticsearch.node.settings.NodeSettingsService.clusterChanged(NodeSettingsService.java:84) > at org.elasticsearch.cluster.service.InternalClusterService$2.run(InternalClusterService.java:321) > at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:95) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > \`\`\` > > So now I have no way to tell what kind of thread pool I have. > > Version: 0.90.3 We had to shut the cluster down, removed the global state file and restarted Elasticsearch. Going forward we'll be using the elasticsearch.yml file to apply these changes. On Wednesday, 4 December 2013 15:51:24 UTC, Jenny Sivapalan wrote: > Hello, > > We've managed to get our cluster into a state where it has pending tasks > and we unable to update cluster settings. > > Has anyone else experienced this and be able to help us to remove these > tasks? We're thinking the alternative option is to kill the cluster and > start again but we'd like to avoid this. > > We got to this state by making changes to the threadpool search queue > size. It's default is 1000 to unbounded (-1) which applied without issue. > We ran a load test on our application. Next we changed the threadpool > search queue size to 500 but this looks like it did not complete although > we can see it has been applied in the :9200/\_cluster/setting. > > The curl command we used to apply the thread pool settings was: > curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { > "threadpool.search.queue\_size" : 500}}' > > Logs for updating the threadpool search queue size from 1000 to -1: > 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] creating > thread\_pool [search], type [fixed], size [40], queue\_size [-1] > 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] processing > [zen-disco-receive(from master [[Cap 'N > Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying updated > cluster\_state (version: 1598) > > Logs for updating the threadpool search queue from -1 to 500 (Note it > never applied.): > 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] creating > thread\_pool [search], type [fixed], size [40], queue\_size [500] > > When we've tried to update the threadpool search queue size again we get > see it being added to the pending tasks list but is then rejected. The > error we get is:- > ProcessClusterEventTimeoutException[failed to process cluster event > (cluster\_update\_settings) within 30s]; ","status":503} > > Looking at the pending tasks (:9200/\_cluster/pending\_tasks) shows > (the cluster\_update\_settings goes in with a higher insert order number > than the ones listed below):- > { > > "tasks": [ > > { > > "insert\_order": 33, > > "priority": "HIGH", > > "source": "update-mapping [][]", > > "time\_in\_queue\_millis": 82081320, > > "time\_in\_queue": "22.8h" > }, > > { > > "insert\_order": 40, > > "priority": "HIGH", > > "source": "update-mapping [][]", > > "time\_in\_queue\_millis": 64831934, > > "time\_in\_queue": "18h" > } > > ] > > } > > We are running Elastic Search 0.90.7. > > Thanks, > Jenny -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [December 5, 2013, 8:31pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/3 "2013-12-05T20:31:17Z")
Hi Jenny, We are running same es version as you and updated cluster transient configuration with no problem. Can you tell what jvm are you running and the version? /Jason On Thu, Dec 5, 2013 at 10:36 PM, Jenny Sivapalan \< [jennifer.sivapalan@gmail.com](mailto:jennifer.sivapalan@gmail.com)\> wrote: > We ended up following the workaround listed on github to resolve this:- > [No obvious way to delete persistent cluster admin settings · Issue #3670 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3670) > > We had to shut the cluster down, removed the global state file and > restarted Elasticsearch. Going forward we'll be using the > elasticsearch.yml file to apply these changes. > > On Wednesday, 4 December 2013 15:51:24 UTC, Jenny Sivapalan wrote: > > > Hello, > > > > We've managed to get our cluster into a state where it has pending tasks > > and we unable to update cluster settings. > > > > Has anyone else experienced this and be able to help us to remove these > > tasks? We're thinking the alternative option is to kill the cluster and > > start again but we'd like to avoid this. > > > > We got to this state by making changes to the threadpool search queue > > size. It's default is 1000 to unbounded (-1) which applied without issue. > > We ran a load test on our application. Next we changed the threadpool > > search queue size to 500 but this looks like it did not complete although > > we can see it has been applied in the :9200/\_cluster/setting. > > > > The curl command we used to apply the thread pool settings was: > > curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { > > "threadpool.search.queue\_size" : 500}}' > > > > Logs for updating the threadpool search queue size from 1000 to -1: > > 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] creating > > thread\_pool [search], type [fixed], size [40], queue\_size [-1] > > 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] > > processing [zen-disco-receive(from master [[Cap 'N > > Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying updated > > cluster\_state (version: 1598) > > > > Logs for updating the threadpool search queue from -1 to 500 (Note it > > never applied.): > > 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] creating > > thread\_pool [search], type [fixed], size [40], queue\_size [500] > > > > When we've tried to update the threadpool search queue size again we get > > see it being added to the pending tasks list but is then rejected. The > > error we get is:- > > ProcessClusterEventTimeoutException[failed to process cluster event > > (cluster\_update\_settings) within 30s]; ","status":503} > > > > Looking at the pending tasks (:9200/\_cluster/pending\_tasks) > > shows (the cluster\_update\_settings goes in with a higher insert order > > number than the ones listed below):- > > { > > > > "tasks": [ > > > > { > > > > "insert\_order": 33, > > > > "priority": "HIGH", > > > > "source": "update-mapping [][]", > > > > "time\_in\_queue\_millis": 82081320, > > > > "time\_in\_queue": "22.8h" > > }, > > > > { > > > > "insert\_order": 40, > > > > "priority": "HIGH", > > > > "source": "update-mapping [][]", > > > > "time\_in\_queue\_millis": 64831934, > > > > "time\_in\_queue": "18h" > > } > > > > ] > > > > } > > > > We are running Elastic Search 0.90.7. > > > > Thanks, > > Jenny > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itydBXjb9mnk2u9mSVYYve3wG6XhbUgU8OywmiaHXR6ErQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itydBXjb9mnk2u9mSVYYve3wG6XhbUgU8OywmiaHXR6ErQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jenny\_Sivapalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenny_sivapalan/32/1884_2.png) [@Jenny\_Sivapalan](https://discuss.elastic.co/u/Jenny_Sivapalan)\ **Post date:** [December 6, 2013, 4:49pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/4 "2013-12-06T16:49:20Z")
Hi Jason, We're running:- java version "1.6.0\_27" OpenJDK Runtime Environment (IcedTea6 1.12.6) (6b27-1.12.6-1ubuntu0.12.10.2) Thanks, Jenny On Thursday, 5 December 2013 20:31:17 UTC, Jason Wee wrote: > Hi Jenny, > > We are running same es version as you and updated cluster transient > configuration with no problem. Can you tell what jvm are you running and > the version? > > /Jason > > On Thu, Dec 5, 2013 at 10:36 PM, Jenny Sivapalan \<[jennifer....@gmail.com](mailto:jennifer....@gmail.com)\ > > > wrote: > > > We ended up following the workaround listed on github to resolve this:- > > [No obvious way to delete persistent cluster admin settings · Issue #3670 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3670) > > > > We had to shut the cluster down, removed the global state file and > > restarted Elasticsearch. Going forward we'll be using the > > elasticsearch.yml file to apply these changes. > > > > On Wednesday, 4 December 2013 15:51:24 UTC, Jenny Sivapalan wrote: > > > > > Hello, > > > > > > We've managed to get our cluster into a state where it has pending tasks > > > and we unable to update cluster settings. > > > > > > Has anyone else experienced this and be able to help us to remove these > > > tasks? We're thinking the alternative option is to kill the cluster and > > > start again but we'd like to avoid this. > > > > > > We got to this state by making changes to the threadpool search queue > > > size. It's default is 1000 to unbounded (-1) which applied without issue. > > > We ran a load test on our application. Next we changed the threadpool > > > search queue size to 500 but this looks like it did not complete although > > > we can see it has been applied in the :9200/\_cluster/setting. > > > > > > The curl command we used to apply the thread pool settings was: > > > curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { > > > "threadpool.search.queue\_size" : 500}}' > > > > > > Logs for updating the threadpool search queue size from 1000 to -1: > > > 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] creating > > > thread\_pool [search], type [fixed], size [40], queue\_size [-1] > > > 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] > > > processing [zen-disco-receive(from master [[Cap 'N > > > Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying > > > updated cluster\_state (version: 1598) > > > > > > Logs for updating the threadpool search queue from -1 to 500 (Note it > > > never applied.): > > > 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] creating > > > thread\_pool [search], type [fixed], size [40], queue\_size [500] > > > > > > When we've tried to update the threadpool search queue size again we get > > > see it being added to the pending tasks list but is then rejected. The > > > error we get is:- > > > ProcessClusterEventTimeoutException[failed to process cluster event > > > (cluster\_update\_settings) within 30s]; ","status":503} > > > > > > Looking at the pending tasks (:9200/\_cluster/pending\_tasks) > > > shows (the cluster\_update\_settings goes in with a higher insert order > > > number than the ones listed below):- > > > { > > > > > > "tasks": [ > > > > > > { > > > > > > "insert\_order": 33, > > > > > > "priority": "HIGH", > > > > > > "source": "update-mapping [][]", > > > > > > "time\_in\_queue\_millis": 82081320, > > > > > > "time\_in\_queue": "22.8h" > > > }, > > > > > > { > > > > > > "insert\_order": 40, > > > > > > "priority": "HIGH", > > > > > > "source": "update-mapping [][]", > > > > > > "time\_in\_queue\_millis": 64831934, > > > > > > "time\_in\_queue": "18h" > > > } > > > > > > ] > > > > > > } > > > > > > We are running Elastic Search 0.90.7. > > > > > > Thanks, > > > Jenny > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [December 6, 2013, 5:15pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/5 "2013-12-06T17:15:23Z")
Hmm... maybe you want to run the oracle jvm. We have a lot in common! We are running jvm version 1.6.0\_25 . Not exactly sure where is your problem but the official doc > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. recommend oracle jvm. You might want to consider switch that , see if the problem arise later day. /Jason On Sat, Dec 7, 2013 at 12:49 AM, Jenny Sivapalan \< [jennifer.sivapalan@gmail.com](mailto:jennifer.sivapalan@gmail.com)\> wrote: > Hi Jason, > > We're running:- > java version "1.6.0\_27" > OpenJDK Runtime Environment (IcedTea6 1.12.6) > (6b27-1.12.6-1ubuntu0.12.10.2) > > Thanks, > Jenny > > On Thursday, 5 December 2013 20:31:17 UTC, Jason Wee wrote: > > > Hi Jenny, > > > > We are running same es version as you and updated cluster transient > > configuration with no problem. Can you tell what jvm are you running and > > the version? > > > > /Jason > > > > On Thu, Dec 5, 2013 at 10:36 PM, Jenny Sivapalan [jennifer....@gmail.com](mailto:jennifer....@gmail.com)wrote: > > > > > We ended up following the workaround listed on github to resolve this:- > > > [No obvious way to delete persistent cluster admin settings · Issue #3670 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3670) > > > > > > We had to shut the cluster down, removed the global state file and > > > restarted Elasticsearch. Going forward we'll be using the > > > elasticsearch.yml file to apply these changes. > > > > > > On Wednesday, 4 December 2013 15:51:24 UTC, Jenny Sivapalan wrote: > > > > > > > Hello, > > > > > > > > We've managed to get our cluster into a state where it has pending > > > > tasks and we unable to update cluster settings. > > > > > > > > Has anyone else experienced this and be able to help us to remove these > > > > tasks? We're thinking the alternative option is to kill the cluster and > > > > start again but we'd like to avoid this. > > > > > > > > We got to this state by making changes to the threadpool search queue > > > > size. It's default is 1000 to unbounded (-1) which applied without issue. > > > > We ran a load test on our application. Next we changed the threadpool > > > > search queue size to 500 but this looks like it did not complete although > > > > we can see it has been applied in the :9200/\_cluster/setting. > > > > > > > > The curl command we used to apply the thread pool settings was: > > > > curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { > > > > "threadpool.search.queue\_size" : 500}}' > > > > > > > > Logs for updating the threadpool search queue size from 1000 to -1: > > > > 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] > > > > creating thread\_pool [search], type [fixed], size [40], queue\_size [-1] > > > > 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] > > > > processing [zen-disco-receive(from master [[Cap 'N > > > > Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying > > > > updated cluster\_state (version: 1598) > > > > > > > > Logs for updating the threadpool search queue from -1 to 500 (Note it > > > > never applied.): > > > > 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] > > > > creating thread\_pool [search], type [fixed], size [40], queue\_size [500] > > > > > > > > When we've tried to update the threadpool search queue size again we > > > > get see it being added to the pending tasks list but is then rejected. The > > > > error we get is:- > > > > ProcessClusterEventTimeoutException[failed to process cluster event > > > > (cluster\_update\_settings) within 30s]; ","status":503} > > > > > > > > Looking at the pending tasks (:9200/\_cluster/pending\_tasks) > > > > shows (the cluster\_update\_settings goes in with a higher insert order > > > > number than the ones listed below):- > > > > { > > > > > > > > "tasks": [ > > > > > > > > { > > > > > > > > "insert\_order": 33, > > > > > > > > "priority": "HIGH", > > > > > > > > "source": "update-mapping [][]", > > > > > > > > "time\_in\_queue\_millis": 82081320, > > > > > > > > "time\_in\_queue": "22.8h" > > > > }, > > > > > > > > { > > > > > > > > "insert\_order": 40, > > > > > > > > "priority": "HIGH", > > > > > > > > "source": "update-mapping [][]", > > > > > > > > "time\_in\_queue\_millis": 64831934, > > > > > > > > "time\_in\_queue": "18h" > > > > } > > > > > > > > ] > > > > > > > > } > > > > > > > > We are running Elastic Search 0.90.7. > > > > > > > > Thanks, > > > > Jenny > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itzHZR4hh%3DMZU9kiSor3mRicKNA\_yt3YzigmE9WLEsQw%3Dw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itzHZR4hh%3DMZU9kiSor3mRicKNA_yt3YzigmE9WLEsQw%3Dw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jenny\_Sivapalan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenny_sivapalan/32/1884_2.png) [@Jenny\_Sivapalan](https://discuss.elastic.co/u/Jenny_Sivapalan)\ **Post date:** [December 6, 2013, 5:55pm UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/6 "2013-12-06T17:55:58Z")
Ah sorry I managed to paste in my local developer box details rather than production version. In production (where we had the issue) we're using java version "1.7.0\_25" Which should be fine. Thanks, Jenny On Friday, 6 December 2013 17:15:23 UTC, Jason Wee wrote: > Hmm... maybe you want to run the oracle jvm. We have a lot in common! We > are running jvm version 1.6.0\_25 . Not exactly sure where is your problem > but the official doc > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/_linux.html#_installing_the_oracle_jdk) recommend > oracle jvm. You might want to consider switch that , see if the problem > arise later day. > > /Jason > > On Sat, Dec 7, 2013 at 12:49 AM, Jenny Sivapalan \<[jennifer....@gmail.com](mailto:jennifer....@gmail.com)\ > > > wrote: > > > Hi Jason, > > > > We're running:- > > java version "1.6.0\_27" > > OpenJDK Runtime Environment (IcedTea6 1.12.6) > > (6b27-1.12.6-1ubuntu0.12.10.2) > > > > Thanks, > > Jenny > > > > On Thursday, 5 December 2013 20:31:17 UTC, Jason Wee wrote: > > > > > Hi Jenny, > > > > > > We are running same es version as you and updated cluster transient > > > configuration with no problem. Can you tell what jvm are you running and > > > the version? > > > > > > /Jason > > > > > > On Thu, Dec 5, 2013 at 10:36 PM, Jenny Sivapalan \<[jennifer....@gmail.com](mailto:jennifer....@gmail.com) > > > > > > > wrote: > > > > > > > We ended up following the workaround listed on github to resolve this:- > > > > [No obvious way to delete persistent cluster admin settings · Issue #3670 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/3670) > > > > > > > > We had to shut the cluster down, removed the global state file and > > > > restarted Elasticsearch. Going forward we'll be using the > > > > elasticsearch.yml file to apply these changes. > > > > > > > > On Wednesday, 4 December 2013 15:51:24 UTC, Jenny Sivapalan wrote: > > > > > > > > > Hello, > > > > > > > > > > We've managed to get our cluster into a state where it has pending > > > > > tasks and we unable to update cluster settings. > > > > > > > > > > Has anyone else experienced this and be able to help us to remove > > > > > these tasks? We're thinking the alternative option is to kill the cluster > > > > > and start again but we'd like to avoid this. > > > > > > > > > > We got to this state by making changes to the threadpool search queue > > > > > size. It's default is 1000 to unbounded (-1) which applied without issue. > > > > > We ran a load test on our application. Next we changed the threadpool > > > > > search queue size to 500 but this looks like it did not complete although > > > > > we can see it has been applied in the :9200/\_cluster/s > > > > > etting. > > > > > > > > > > The curl command we used to apply the thread pool settings was: > > > > > curl -XPUT :9200/\_cluster/settings -d '{ "transient" : { > > > > > "threadpool.search.queue\_size" : 500}}' > > > > > > > > > > Logs for updating the threadpool search queue size from 1000 to -1: > > > > > 2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] > > > > > creating thread\_pool [search], type [fixed], size [40], queue\_size [-1] > > > > > 2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] > > > > > processing [zen-disco-receive(from master [[Cap 'N > > > > > Hawk][1SA1x8JwRHCk5xm9\_MmLGA][inet[:9300]]])]: done applying > > > > > updated cluster\_state (version: 1598) > > > > > > > > > > Logs for updating the threadpool search queue from -1 to 500 (Note it > > > > > never applied.): > > > > > 2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] > > > > > creating thread\_pool [search], type [fixed], size [40], queue\_size [500] > > > > > > > > > > When we've tried to update the threadpool search queue size again we > > > > > get see it being added to the pending tasks list but is then rejected. The > > > > > error we get is:- > > > > > ProcessClusterEventTimeoutException[failed to process cluster event > > > > > (cluster\_update\_settings) within 30s]; ","status":503} > > > > > > > > > > Looking at the pending tasks (:9200/\_cluster/pending\_tasks) > > > > > shows (the cluster\_update\_settings goes in with a higher insert order > > > > > number than the ones listed below):- > > > > > { > > > > > > > > > > "tasks": [ > > > > > > > > > > { > > > > > > > > > > "insert\_order": 33, > > > > > > > > > > "priority": "HIGH", > > > > > > > > > > "source": "update-mapping [][]", > > > > > > > > > > "time\_in\_queue\_millis": 82081320, > > > > > > > > > > "time\_in\_queue": "22.8h" > > > > > }, > > > > > > > > > > { > > > > > > > > > > "insert\_order": 40, > > > > > > > > > > "priority": "HIGH", > > > > > > > > > > "source": "update-mapping [][]", > > > > > > > > > > "time\_in\_queue\_millis": 64831934, > > > > > > > > > > "time\_in\_queue": "18h" > > > > > } > > > > > > > > > > ] > > > > > > > > > > } > > > > > > > > > > We are running Elastic Search 0.90.7. > > > > > > > > > > Thanks, > > > > > Jenny > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/473ab223-68f9-4866-a98f-cb724ad06197% > > > > > [40googlegroups.com](http://40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e77caf15-242a-44d1-8a25-41dadee1e3b3%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da6971b2-98c1-49c7-8e28-b1873f1fe50a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da6971b2-98c1-49c7-8e28-b1873f1fe50a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:02am UTC](https://discuss.elastic.co/t/cluster-has-pending-tasks-cannot-update-cluster-settings/14706/7 "2017-07-06T02:02:48Z")