Hi,
35 threads are not particularly much for a server process. From the thread dump I just see that a few HTTP handler threads are allocating buffer memory (presumably for incoming bulk requests). I suggest that you take a closer look at the heap dump instead (for example with Memory Analyzer which should reveal why the client nodes need so much memory. I guess it is because you're sending / buffering too much bulk index traffic for the amount of memory you've allocated. This means you either need to increase the system capacity or reduce the load on the client nodes. So you can try to:
- Reduce bulk request size
- Reduce the number of clients sending bulk index requests
- Reduce incoming queue lengths
- Increase heap memory on the client nodes
- Add more data nodes (based on the premise that they cannot cope with the amount of traffic sent and they put backpressure on the client nodes)
But these are just pointers based on my assumption above. You need to check the heap dump and run further tests. I hope this gets you started.
Daniel