How to balance my elasticsearch cluster

Elastic Stack Elasticsearch
1

Hello,

I want to balance my Elasticsearch cluster.

I am on version 7.16.3

I have 5 servers on cluster, but the data are not use symmetrically :

ELS01 : 62%
ELS02 : 50%
ELS03 : 69%
ELS04 : 76%
ELS05 : 86%

This is my configuration :

cluster.name: ELS-Cluster
cluster.initial_master_nodes: 192.168.1.61

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.key: /etc/elasticsearch/certs/ELS01.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/ELS01.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ELS.ca.crt" ]

node.name: node-1

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/ELS01.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/ELS01.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ELS.ca.crt"  ]
xpack.security.transport.ssl.client_authentication: optional

path.data: /data/ELScommon
path.logs: /var/log/elasticsearch

network.host: 192.168.1.61

discovery.seed_hosts: ["192.168.1.61", "192.168.1.62", "192.168.1.63", "192.168.1.64", "192.168.1.65"]

Is exist easy manipulation to balance this cluster ?

Thank you.

1 Like
2

Hello @Arethusa

Yes, it does exist with fill affinity and anti affinity.

For more information follow this for implementation

3

That's for ECE, not Elasticsearch.

1 Like
4

Do you have the same shard count on each node? _cat/nodes?V will show you.

1 Like
5

I have 5 servers on cluster, but the data are not use symmetrically :

ELS01 : 62%
ELS02 : 50%
ELS03 : 69%
ELS04 : 76%
ELS05 : 86%

I wonder why this seems problematic to you. This is perfectly normal Elasticsearch behavior. It does not balance by disk space, rather it uses the low and high watermarks to ensure that no node gets to a full disk.

As long as your workload is served well, this is no problem. Perhaps there is an underlying symptom you are chasing?

1 Like
6

Hello everyone, thank you for your answers.

@warkolm Below my shard repartitions:

ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role   master name
192.168.1.63           65          90  50    0.52    1.18     1.00 cdfhilmrstw *      ELS03
192.168.1.64           32          93  35    0.12    0.35     0.42 cdfhilmrstw -      ELS04
192.168.1.61           66          99  24    1.32    1.76     1.83 cdfhilmrstw -      ELS01
192.168.1.62           60          99  21    0.23    0.55     0.56 cdfhilmrstw -      ELS02
192.168.1.65           46          99  95    2.99    3.08     3.21 cdfhilmrstw -      ELS05

@HenningAndersen

Than means my ELS05 can't go 100 % while the other nodes are not at 100 % too ?

Thank you.

7

@Arethusa , does that mean that the percentages you showed in your original posting was cpu-usage? If so, is that persistent over the course of a day? Does it slow down some of your searches or indexing?

Elasticsearch does not balance by cpu-usage, but it does have configurable rules that can help the balancer. However, this typically constraints the cluster from balancing and moving shards freely, which can lead to resilience issues if not used carefully. So before embarking on such tuning, it is worth ensuring that there is a client-visible issues to resolve?

8

Hello @HenningAndersen
The original posting is about data usage, not the CPU usage.

Thank you for reply.

9

@Arethusa ,

is there a problem you need to solve here and if so, what problem is that? Or is this more of a learning/playing around exercise?

10

After rereading it is actually not very clear. Sorry for that.
I'm talking about data balancing, not CPU or whatever.

So yeah, I'm afraid to see one ELS near 90% data usage, and another near 50% data usage.

So I would have liked if possible to be able to balance the data on the 5 nodes.

Thank you.

11

@Arethusa thanks for clarifying.

I'm afraid to see one ELS near 90% data usage, and another near 50% data usage.

This is perfectly normal behavior as per my prior comment. If a node grows to 90%+ disk usage, Elasticsearch will move shards off the node to nodes below 85% disk usage. Those limits can be controlled via the watermark settings, but are typically left to their defaults.

As long as shards can be moved elsewhere you should thus never hit 100% disk usage on a single node (but could temporarily see nodes go slightly over 90% disk usage).

2 Likes
12

Hello,

Your indices shard setting may be the source of this. If you have only one shard per index the data of this index won't be balanced between hosts that's why you may have hosts using more spaces than others

Data are balanced by shards, and one shard only sits on one host (you may have a replica of it on another host though). Let say you have a big index whith only one shard ,no replica and 4 smaller indices. Each host will store each index but the one which hosts the big index will have a much higher disk usage than the others.

Take what i said with a grain of salt as it's based on my understanding of how es works.

2 Likes
13

Ok, it's clear for me.

Thank you for you time and availability.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.