High number of refresh threads

Elastic Stack Elasticsearch
1

Hello,
Here is the result of
GET /_cat/thread_pool/refresh/?v

node_name name active queue rejected
node-5203 refresh 0 0 0
node-5202 refresh 10 13676 0
node-5206 refresh 0 0 0
node-5201 refresh 10 75341 0
node-5205 refresh 1 0 0
node-5204 refresh 10 3494 0

I have 6 nodes, and some of them have a huge number of refresh in queue.
How can i get rid of them ?
I already change the refresh_interval from 10s to 60s, but i guess refresh in queue are now useless. is there a way to delete them ?
Thanks

2

Does the system load avg high in the nodes?
And How the io util grows to each node when this happen?

3

What type of storage do you have? Local SSDs? Are all nodes the same specification? Do you have any indices on these nodes for which you perform a lot of updates and/or use a lot of nested documents?

4

Hello.
Figured it out. A huge number of Spark threads updating the same document with a refresh_interval set to 1sec... This was a misconfiguration where the elasticsearch document_id was supposed to be different. Therefore instead of creating a huge number of different douments in ES, the cluster tried to update only one.