Need some help to understand high memory issue we have and possibly fixed/improve our elastic stack setup (without adding new server). We were given only 2 servers to install elasticsearch and we must set it up in high availability. Each server has 32GB RAM and 16CPU and we do not have Kibana and Logtash running in these servers.
ES instance1 node.roles: [ data, master ]
ES instance2 node.roles: [ master, voting_only ]
ES instance3 node.roles: [ data, master ]
ES instance4 node.roles: [ master, voting_only ]
( total of 4 elasticsearch instances in the cluster)
We decided to have both master-data and master-voter_only in one server so we can avoid cluster going down when one server is down. But we are observing high memory utilization from voter_only instances.
If both instances (master-data and master-voter_only) are enabled in a server, the memory utilization gets 93%. If we turned off master-voter_only instance, the memory utilization becomes 40%. Any idea why voter_only instance is taking 50% memory usage? We tried changing the JVM Heap size in jvm.options file to 4GB or 8GB but it did not improve anything.