Hello, I am currently setting up an Elasticsearch cluster with three nodes and encountering an issue with cluster formation. Here's the setup:
- elastic01: 10G network interface with VLAN configured, able to join the cluster.
- elastic02: 10G network interface with VLAN configured, able to join the cluster.
- elastic03: 1G network interface without VLAN, unable to join the cluster.
My elasticsearch.yml configuration for the third node (elastic03) is as follows:
cluster.name: escluster
node.name: elastic03
path.repo: /backup/elasticsearch
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["<<node1 ip>>:9300","<<node2 ip>>:9300","<<node3 ip>>:9300"]
cluster.initial_master_nodes: ["elastic01","elastic02","elastic03"]
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs/http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: certs/transport.p12
xpack.security.transport.ssl.truststore.path: certs/transport.p12
http.host: 0.0.0.0
transport.host: 0.0.0.0
However, I am facing an issue where the third node (elastic03) is unable to discover the master node. The error message is as follows:
[2024-02-24T16:31:59,455][INFO ][o.e.n.Node ] [elastic03] starting ...
[2024-02-24T16:31:59,463][INFO ][o.e.x.s.c.f.PersistentCache] [elastic03] persistent cache index loaded
[2024-02-24T16:31:59,464][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [elastic03] deprecation component started
[2024-02-24T16:31:59,523][INFO ][o.e.t.TransportService ] [elastic03] publish_address {203.145.195.168:9300}, bound_addresses {[::]:9300}
[2024-02-24T16:31:59,601][INFO ][o.e.b.BootstrapChecks ] [elastic03] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-02-24T16:31:59,604][INFO ][o.e.c.c.ClusterBootstrapService] [elastic03] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [elastic01, elastic02, elastic03]
[2024-02-24T16:32:09,609][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elastic03] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [elastic01, elastic02, elastic03] to bootstrap a cluster: have discovered [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}]; discovery will continue using [203.145.195.166:9300, 203.145.195.167:9300] from hosts providers and [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-24T16:32:19,610][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elastic03] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [elastic01, elastic02, elastic03] to bootstrap a cluster: have discovered [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}]; discovery will continue using [203.145.195.166:9300, 203.145.195.167:9300] from hosts providers and [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-24T16:32:29,611][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elastic03] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [elastic01, elastic02, elastic03] to bootstrap a cluster: have discovered [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}]; discovery will continue using [203.145.195.166:9300, 203.145.195.167:9300] from hosts providers and [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-24T16:32:29,611][WARN ][o.e.n.Node ] [elastic03] timed out while waiting for initial discovery state - timeout: 30s
[2024-02-24T16:32:29,615][INFO ][o.e.h.AbstractHttpServerTransport] [elastic03] publish_address {203.145.195.168:9200}, bound_addresses {[::]:9200}
[2024-02-24T16:32:29,622][INFO ][o.e.n.Node ] [elastic03] started {elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}{ml.allocated_processors_double=64.0, ml.allocated_processors=64, ml.machine_memory=83546791936, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=12.0.0, ml.max_jvm_size=33285996544}
[2024-02-24T16:32:39,612][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elastic03] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [elastic01, elastic02, elastic03] to bootstrap a cluster: have discovered [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}]; discovery will continue using [203.145.195.166:9300, 203.145.195.167:9300] from hosts providers and [{elastic03}{gl8lpMk2SW-trYTIj5m0VA}{SXSTWNt0SQ-g8EWbaIgfcw}{elastic03}{203.145.195.168}{203.145.195.168:9300}{cdfhilmrstw}{8.12.1}{7000099-8500010}] from last-known cluster state; node term 0, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
Why might this error be occurring, and how can I resolve it to successfully form the cluster?
Thank you for your assistance.