Thread Pool stats show queued items despite not being full

Elastic Stack Elasticsearch
1

Good day,

in our monitoring data I can see that one cluster with search load (dozens of queries per second) shows slightly confusing thread pool stats.

I can regularly see searches being queued while the thread pool size is never reached when running the cat thread pool endpoint.

See this example output of the nodes thread pool API (cut down to the needed fields):

"search": {
  "threads": 13,
  "queue": 4,
  "active": 6
}

another one

"search": {
  "threads": 13,
  "queue": 3,
  "active": 4
}

I’ve never seen the active value at its upper limit of the thread pool size, when there were queued threads, but always something like 4,5 or 6.

My only assumption here is, that due to fast running queries the thread pool is draining faster than the queue. Is that a solid assumption?

This is on 8.18.3.

Thanks and have a great week!

–Alex