Hello Community,
I have an elk cluster with 4 servers 1master and 3 worker nodes. My issue is with the disk space consumption on worker nodes with regards to data indexing and replicas. When an index is created one replica is stored on master and one on worker nodes since for each index the number of replicas is 2. In recent times,, I have been noticing that a majority of index replicas are being stored on worker node1 which is causing disk space issues. Out of three worker nodes my worker node1 is always with low disk space while other two nodes have ample disk space. My question here is how can I balance these replicas equally on all three worker nodes?
Terminology is not quite right,, can you send output from GET on _cluster/health, which will look something like:
curl -s -k -u "${EUSER}":"${EPASS}" "https://${EHOST}:${EPORT}/_cluster/health" | jq .
{
"cluster_name": "name",
"status": "green",
"timed_out": false,
"number_of_nodes": 5,
"number_of_data_nodes": 5,
...
ES tries to "balance" things itself, and maybe it has done so, but it cannot know a priori how much data each shard/index will get. You want to look at
There is settings to control the rebalancing of shards across the cluster.
Enable these setting in the cluster, for this activity the rolling restart of the cluster is required.
One more issue with single master, being single master can cause the cluster desired state fi the master is restarted. Can't you have all your nodes master and data eligible.