We run a 10 data nodes cluster, 64Gb memory, 30 Gb heap, 8 cores, with elasticsearch 2.4. At least 500 searchs/seconds.
Every 3 days, we see errors because the search threads pool is full, after some investigations, the problem appears when the node is doing a GC, at 70% of heap size => 10%. It took some seconds but this is enough to cause trouble.
Is it possible to tell to the client node to not sending shard search to a node when the node is doing a GC ? (or when the heap is at X % so the node is going to GC)