This isn't a huge deal, but wanted to point it out to see if there
are any ways to avoid. Current production load is pretty low, 40 or so
queries per second max. Performance is great and we know we can ramp
up the traffic to 25x this rate. Average response times are ~25ms and
max are ~200ms.
However, we have seen two performance hits, where averages jumped to a
sec and maxs up to 12 seconds. Both times this has occurred it
directly correlates one of the machines in the cluster hitting its JVM
heap max for elasticsearch. After the limit has been hit, each machine
has been fine.
Any ideas if there is anyway to avoid this hit? I'm guessing that the
GC kicks on and ends up blocking some processing.