Stop routing search when the node is GC

We run a 10 data nodes cluster, 64Gb memory, 30 Gb heap, 8 cores, with elasticsearch 2.4. At least 500 searchs/seconds.

Every 3 days, we see errors because the search threads pool is full, after some investigations, the problem appears when the node is doing a GC, at 70% of heap size => 10%. It took some seconds but this is enough to cause trouble.

Is it possible to tell to the client node to not sending shard search to a node when the node is doing a GC ? (or when the heap is at X % so the node is going to GC)

No it's not, a client has no idea of that. Even if it knew a node's heap use, it doesn't know if it's in GC or will go into GC for the query it sends.

So the balance is totally random?

I am thinking to a new component such as "routing strategy" , so we could implement different balance strategies.

What balance do you mean?

I wonder if adaptive replica selection would help here? It would however require that you upgrade...

As far as I know there is no way to control this in earlier versions.

1 Like

what I called "balance" is how the client node performs the shard search (if I understood how search is working ^^ APP --> CLIENT(index) --> DATA(shards))

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.