Hello,
We've managed to get our cluster into a state where it has pending tasks
and we unable to update cluster settings.
Has anyone else experienced this and be able to help us to remove these
tasks? We're thinking the alternative option is to kill the cluster and
start again but we'd like to avoid this.
We got to this state by making changes to the threadpool search queue size.
It's default is 1000 to unbounded (-1) which applied without issue. We ran
a load test on our application. Next we changed the threadpool search queue
size to 500 but this looks like it did not complete although we can see it
has been applied in the :9200/_cluster/setting.
The curl command we used to apply the thread pool settings was:
curl -XPUT :9200/_cluster/settings -d '{ "transient" : {
"threadpool.search.queue_size" : 500}}'
Logs for updating the threadpool search queue size from 1000 to -1:
2013-12-03 14:04:44,515 DEBUG threadpool [X-Man] creating
thread_pool [search], type [fixed], size [40], queue_size [-1]
2013-12-03 14:04:44,516 DEBUG cluster.service [X-Man] processing
[zen-disco-receive(from master [[Cap 'N
Hawk][1SA1x8JwRHCk5xm9_MmLGA][inet[:9300]]])]: done applying updated
cluster_state (version: 1598)
Logs for updating the threadpool search queue from -1 to 500 (Note it never
applied.):
2013-12-03 15:13:32,901 DEBUG threadpool [X-Man] creating
thread_pool [search], type [fixed], size [40], queue_size [500]
When we've tried to update the threadpool search queue size again we get
see it being added to the pending tasks list but is then rejected. The
error we get is:-
ProcessClusterEventTimeoutException[failed to process cluster event
(cluster_update_settings) within 30s]; ","status":503}
Looking at the pending tasks (:9200/_cluster/pending_tasks) shows
(the cluster_update_settings goes in with a higher insert order number than
the ones listed below):-
{
"tasks": [
{
"insert_order": 33,
"priority": "HIGH",
"source": "update-mapping [][]",
"time_in_queue_millis": 82081320,
"time_in_queue": "22.8h"
},
{
"insert_order": 40,
"priority": "HIGH",
"source": "update-mapping [][]",
"time_in_queue_millis": 64831934,
"time_in_queue": "18h"
}
]
}
We are running Elastic Search 0.90.7.
Thanks,
Jenny
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7d2767ed-1256-470c-9825-f34e63a28a2c%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.