Thread_pool settings have strange defaults in es v6.2.4

Elastic Stack Elasticsearch
1

In the elasticsearch.yml config I'm setting the search thread_pool size to be 4200, like so:

thread_pool.search.queue_size: 4200

But the other day I received the following error:

ElasticsearchStatusException[Elasticsearch exception [type=search_phase_execution_exception, reason=]]; nested: 
  ElasticsearchException[Elasticsearch exception [
    type=es_rejected_execution_exception, 
    reason=rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@4993756d on 
    QueueResizingEsThreadPoolExecutor[
      name = es-live-7/search, 
      queue capacity = 1000, 
      min queue capacity = 1000, 
      max queue capacity = 1000, 
      frame size = 2000, 
      targeted response rate = 1s, 
      task execution EWMA = 10.5ms, 
      adjustment amount = 50, 
 org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@28b39f6f[Running, 
  pool size = 13, 
  active threads = 13, 
  queued tasks = 1088, 
  completed tasks = 89917206]]]];

Why is it reporting the queue capacity as 1000 and not 4200?
I know that 1000 is the default, so did my config not work?
How can I enforce a minimum queue size of 4200? Do I need to set:

thread_pool.search.min_queue_size: 4200

as well?
Or do I just need to change the type of queue to fixed?
If so, where is the documentation on setting that config?

I just executed GET /_cluster/settings?include_defaults=true
then I get this:

"search": {
                "max_queue_size": "1000",
                "queue_size": "4200",
                "size": "13",
                "auto_queue_frame_size": "2000",
                "target_response_time": "1s",
                "min_queue_size": "1000"
            },

Why would the max_queue_size default to be above the queue_size?

I think what I need to do is change my configuration to:

thread_pool:
  search:
    min_queue_size: 4200
    max_queue_size: 4200
    queue_size: 4200

Can someone please confirm?
Sorry for the rambling post!

Thanks!

2

The documentation tell us that the search thread pool is of type fixed_auto_queue_size and has the following to say about its parameters (emphasis mine):

The queue_size allows to control the initial size of the queue of pending requests that have no threads to execute them.

The min_queue_size setting controls the minimum amount the queue_size can be adjusted to.

The max_queue_size setting controls the maximum amount the queue_size can be adjusted to.

You have set the initial size to be above the maximum size, so the queue shrinks almost immediately to the default maximum of 1000.

3

Thanks! Missed that in my obvious skimming. Time to cycle 10 es nodes, then.

4

Quick question: if this is a cluster-wide setting, and I roll in a new node with a different setting...what happens?

5

These settings are not cluster-wide, they're local to each node. It's probably not the best idea to have them set differently across different nodes, but there's no technical constraint that prevents you from doing so.

1 Like
6

Ah yes, I see that when I deploy a new node with the different settings and run _cluster/settings against it the values are different.

Note to self: _cluster/settings does not show cluster-wide settings.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.