I recently changed thread_pool type of "bulk" from the "cached" (default) to "blocking".
This was because I observed very high number of threads (about 20,000) and often times it ended up with losing the node. (by slow response of the node).
I thought the number of thread itself was choking the system, so I'm trying the blocking type.
Since changing it to the blocking, the system is stable. (so far. I changed it only a couple of days ago.)
By the way, the question here is about node stats.
The curl localhost:9200/_cluster/nodes//stats?thread_pool=true gives us
"threads", "queue", "active" for each thread pool.
I understand "queue" is the number of tasks waiting in the thread pool queue.
What I don't understand is "threads" and "active".
My assumption is "thread" is the number of threads created in the thread pool and "active" is the number of thread which is currently running.
Is this correct?
It seems "threads" is simply growing. Does it decrease some time later in some condition?
Are there difference by the type of the thread pool?
I can imagine "queue" has nothing to do with "cached", "threads" do not change with "fixed".
So the meaningful measure for "cached" is "threads" (or "active"?) and for "fixed" is "queue" and "active"?
I'm trying to monitor the node's 'busyness' by monitoring the thread_pool in node stats.
Thanks for your help.