Different load on nodes

(Matija Bruncic) #1


We had more load on our cluster than usual and at that time we noticed some strange behavior. A node that hold primaries had much higher GC count. All nodes have the same configuration, JVM: 1.8.0_162 ES: 5.6.4. The only difference between node #1 and node #4 was that #1 had 2 primaries while #4 had 2 replicas. All shards have roughly the same size (~5GB).
Is it possible that node that holds primaries have higher memory consumption that one that holds only replicas?

Screenshots from cerebro and grafana:
New photo by Matija Bruncic
New photo by Matija Bruncic

(Christian Dahlqvist) #2

Are you performing lots of updates, e.g. using scripts?

(Matija Bruncic) #3

Hi Christian,

That's correct, we use scripted update in each request

(Christian Dahlqvist) #4

Scripted updates are performed on the primary, and the resulting document is then replicated, meaning that the script processing is likely to result in more GC on the node with more primary shards.

(Matija Bruncic) #5

Great, thx. That explains a lot.
Is there maybe a feature to "force" even distribution of primary shards across the nodes? In our case, I'd like to have primary for shard #3 on node #4. It doesn't have to be automatic, maybe we can add a feature in our service to watch for that kind of state and manually promote replica shards

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.