Rejected thread pool

[jdambly (Jeff d'Ambly)]

I am getting rejected thread pools on just one node, how can I resolve this?

node_name            name                active rejected completed 
elastic-es-data-2    analyze                  0        0         0
elastic-es-data-2    ccr                      0        0         0
elastic-es-data-2    data_frame_indexing      0        0         0
elastic-es-data-2    fetch_shard_started      0        0         0
elastic-es-data-2    fetch_shard_store        0        0         1
elastic-es-data-2    flush                    0        0       144 
elastic-es-data-2    force_merge              0        0         0
elastic-es-data-2    generic                  0        0     14745
elastic-es-data-2    get                      0        0         6
elastic-es-data-2    listener                 0        0        14
elastic-es-data-2    management               1        0     29589
elastic-es-data-2    ml_datafeed              0        0         0
elastic-es-data-2    ml_job_comms             0        0         0
elastic-es-data-2    ml_utility               0        0         0
elastic-es-data-2    refresh                  0        0     17059
elastic-es-data-2    rollup_indexing          0        0         0
elastic-es-data-2    search                   0        0      2015
elastic-es-data-2    search_throttled         0        0         0
elastic-es-data-2    security-token-key       0        0         0
elastic-es-data-2    snapshot                 0        0         0
elastic-es-data-2    warmer                   0        0         0
elastic-es-data-2    watcher                  0        0         0
elastic-es-data-2    write                    1    15129     50417

[sebgl (Sebastien Guilloux)]

Hi @jdambly, this looks like an Elasticsearch question more than an ECK question.
I suggest you ask it in https://discuss.elastic.co/c/elasticsearch/6 instead.

[DavidTurner]

The write threadpool is used to index documents, so it looks like this node cannot keep up with the indexing load.

There's not a lot of information to go on here, but two common issues that can cause this are that the node is faulty (e.g. it has slow or failing disks) or is doing more than its fair share of work (e.g. it has more shards of "hot" indices than the other nodes).

[jdambly (Jeff d'Ambly)]

ok sure, how do you decide what the thread pool should be in k8s? if create a resource pool of say 4 cpus, should I also set the ES config to match?

[DavidTurner]

Elasticsearch will by default set the size of its threadpools appropriately, and it's unlikely that you need to change that.

[jdambly (Jeff d'Ambly)]

ok, so assuming the disk cannot keep up with all the writes, (I'm just writing logs) how can I quantify that?

[DavidTurner]

One way would be to would look at tools like iostat that report things from the OS's point of view, and compare this to your other better-performing nodes.