We had more load on our cluster than usual and at that time we noticed some strange behavior. A node that hold primaries had much higher GC count. All nodes have the same configuration, JVM: 1.8.0_162 ES: 5.6.4. The only difference between node #1 and node #4 was that #1 had 2 primaries while #4 had 2 replicas. All shards have roughly the same size (~5GB).
Is it possible that node that holds primaries have higher memory consumption that one that holds only replicas?
Screenshots from cerebro and grafana: