we have several nodes in our elasticsearch cluster( each shard have only 1 replica though ), but even if one of them call gc i see that qtime is increasing and our web app responce time as well. i think we use default gc settings. elastic version is 2.3.3
Hi @eprst,
isn't it expected that query processing (or any operation for that matter) slows down if the GC kicks in? After all, it needs to stop all JVM threads (at least in some GC phases).
You should check the section on heap sizing in the Definitive Guide. Also standard GC tuning advice for JVM-based applications applies.
To test your setting changes see "Test Performance Scientifically" in the Definitive Guide and I can recommend Rally for helping you with that (although I may be biased as I am the main author of Rally ;)).
Daniel
yes it is kind of obvious, but on the other hand we have 6 nodes, so it should not really slow down the entire app.
Hi,
we have 6 nodes, so it should not really slow down the entire app.
Maybe I misunderstand you but how do you arrive at this conclusion?
Suppose you have N nodes, no replicas and M shards (I just want to keep replicas out of the example to make it a bit simpler).
When you issue a search request, it will arrive at some node in the cluster. This node is called the "coordinating node" as it coordinates all the work that needs to be done to answer your query. The coordinating node will forward the search request to all shards (let's assume that shards are also distributed across all nodes). So the query will hit each node. If just one of them experiences a long GC pause, this will affect the overall duration of the query (This applies similarly if you have replicas).
Daniel