Dynamic queue sizes

Elastic Stack Elasticsearch
1

I wanted to hear thoughts on adding an API to make queues dynamic. I understand this is an advanced setting and there can be drawbacks to raising queue sizes etc. but a queue does help during bursty behaviors and even throughput in certain scenarios

On earlier versions, there used to be dynamic cluster settings to control this but it got removed with 5.x as clusters become heterogeneous node types with eg. hot/cold hardware and having a single dynamic cluster setting to control queues does not make sense.

Do you see value in keeping these dynamic at a node level? There could be attribute driven APIs to tweak them?

2

I will be happy to contribute a PR if such a feature might be interesting to broader users.

3

Are you talking about threadpools?

If so they are dynamic, in that they are based on the number of CPU threads on the system at start up and the max value that you can define.

4

Yes, like you said today these thresholds can be defined on the yml and the node be restarted.

By dynamic - what i am proposing is to do these through a settings API without requiring node restarts.

5

I asked around about this and got some extra info;

We removed the ability to dynamically change thread pool settings; most production clusters don’t have a homogenous set of nodes; there’s dedicated master nodes and hot vs. warm nodes and dedicated ML nodes and coordinating nodes which all might run on different underlying hardware, so it didn’t make sense for this to be cluster wide setting and thus it doesn’t make sense for it to be updatable via the api

Also related - https://github.com/elastic/elasticsearch/pull/18674

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.