Trouble setting up an elasticsearch cluster

Hi, i'm trying to set up a cluster with 3 nodes

es-node1
es-node2
es-node3

installed elasticsearch 7.4 on 3 nodes con configure elasticsearch.yml with the following:

cluster.name: es-cluster
node.name: es-node1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 0.0.0.0
discovery.seed_hosts: ["es-node1"]
cluster.initial_master_nodes: ["es-node1"]

however when i start nodes each node starts its own cluster:

[root@es-node1 elasticsearch]# curl -XGET http://192.168.90.225:9200/_cluster/health?pretty
{
  "cluster_name" : "es-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 85,
  "active_shards" : 85,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 44,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 65.89147286821705
}

Selinux is in permissive and firewall down to avoid trouble.

Any idea?

Finally made it work, had to start each node with a slightliy different configuration

node1:
cluster.name: es-cluster
node.name: es-node1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.225
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1

node2: 
cluster.name: es-cluster
node.name: es-node2
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.226
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1
 - es-node2

node3:
cluster.name: es-cluster
node.name: es-node3
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.227
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1
 - es-node2
 - es-node3

For the benefit of future readers, the solution described above contains a misconfiguration which can result in data loss. From the docs:

WARNING: You must set cluster.initial_master_nodes to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping...

This is not what you have done.

1 Like

Thanks David, but with all nodes in cluster.initial_master_nodes cluster did not start.
However in the way i managed to get it started now i see all nodes forming the cluster:

curl --cacert /etc/elasticsearch/certs/ca.crt -u elastic -XGET https://192.168.90.225:9200/_cat/nodes?v
Enter host password for user 'elastic':
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.90.226           14          21   0    0.00    0.01     0.05 dilm      -      es-node2
192.168.90.227            9          21   0    0.00    0.01     0.05 dilm      *      es-node3
192.168.90.225            8          21   0    0.00    0.01     0.05 dilm      -      es-node1

So i guess i got a single cluster, what else do you think might have been wrong?

thanks

Fortunately it looks like you did indeed get a single cluster, but this was luck. You could have ended up with two.

All three nodes is the correct setting for cluster.initial_master_nodes so I suspect something else was going wrong there. Do you have logs from the case where the cluster did not start? If so can you share them here?

Unfortunatelly i dont have logs, whenever it did not work i deleted the logs prior to some modification to avoid having to dig to much into the logs on the next try.

regards

Ok, not to worry. If you manage to reproduce it again then please get back in touch and we'll try and work out what's happening.