Thread_pool settings have strange defaults in es v6.2.4

In the elasticsearch.yml config I'm setting the search thread_pool size to be 4200, like so: 4200

But the other day I received the following error:

ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=]]; nested: 
  ElasticsearchException[Elasticsearch exception [
    reason=rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@4993756d on 
      name = es-live-7/search, 
      queue capacity = 1000, 
      min queue capacity = 1000, 
      max queue capacity = 1000, 
      frame size = 2000, 
      targeted response rate = 1s, 
      task execution EWMA = 10.5ms, 
      adjustment amount = 50, 
  pool size = 13, 
  active threads = 13, 
  queued tasks = 1088, 
  completed tasks = 89917206]]]];

Why is it reporting the queue capacity as 1000 and not 4200?
I know that 1000 is the default, so did my config not work?
How can I enforce a minimum queue size of 4200? Do I need to set: 4200

as well?
Or do I just need to change the type of queue to fixed?
If so, where is the documentation on setting that config?

I just executed GET /_cluster/settings?include_defaults=true
then I get this:

"search": {
                "max_queue_size": "1000",
                "queue_size": "4200",
                "size": "13",
                "auto_queue_frame_size": "2000",
                "target_response_time": "1s",
                "min_queue_size": "1000"

Why would the max_queue_size default to be above the queue_size?

I think what I need to do is change my configuration to:

    min_queue_size: 4200
    max_queue_size: 4200
    queue_size: 4200

Can someone please confirm?
Sorry for the rambling post!


The documentation tell us that the search thread pool is of type fixed_auto_queue_size and has the following to say about its parameters (emphasis mine):

The queue_size allows to control the initial size of the queue of pending requests that have no threads to execute them.

The min_queue_size setting controls the minimum amount the queue_size can be adjusted to.

The max_queue_size setting controls the maximum amount the queue_size can be adjusted to.

You have set the initial size to be above the maximum size, so the queue shrinks almost immediately to the default maximum of 1000.

Thanks! Missed that in my obvious skimming. Time to cycle 10 es nodes, then.

Quick question: if this is a cluster-wide setting, and I roll in a new node with a different setting...what happens?

These settings are not cluster-wide, they're local to each node. It's probably not the best idea to have them set differently across different nodes, but there's no technical constraint that prevents you from doing so.

1 Like

Ah yes, I see that when I deploy a new node with the different settings and run _cluster/settings against it the values are different.

Note to self: _cluster/settings does not show cluster-wide settings.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.