Hello. I have an ES cluster consisting of 3 hosts and each host is running 8 datanodes + 1 master node, a total of 27 nodes (running as docker containers) plus one separate active master node as a VM in a separate network. I have separated the nodes logically based on the host they are located on with "cluster.routing.allocation.awareness.attributes=hostX/Y/Z".
We have observed that when activating lots of Security Rules in Kibana then one of three hosts has a substantially higher CPU load (15%) than the other hosts (4%). "Docker stats" shows constant 300%+ CPU usage for each node on the one host while the other hosts have a ~50% load on each node.
The active master node is not on one of the three hosts but on the VM on separate hardware.
Is this normal behavior that one host and all the nodes on that host have a lot higher CPU usage than the other hosts? It would be nice to see the CPU load balanced over all three hosts.