My es cluster is 8 identical nodes being fed web logs from Logstash. All of the nodes are listed in the Logstash output so I am assuming that they are equally bearing the import load.
Each node has 64GBytes RAM with the ES Heap set to the recommended 30GBytes
The java heap usage varies from 40% on lowest jvm usage node to 70% (or more) on the highest usage node. I usually have one or two nodes about 70%, 3 or 4 nodes between 50% and 67% java heap usage and 2 or 3 between 30% and 45%. My Master node is almost always in this bottom group.
Knowing that the java heap goes into garbage collection at 75%, I'd like to even out the java heap usage across all nodes. Or at minimum bring the top and bottom usage ones into the middle ground.
Does anyone have any suggestions about how to balance this java heap usages or some tools I can use to see why the same nodes are always in the top group using almost 75%? I have looked at these top nodes and do not see anything different running on them that the other nodes.