Trouble setting up an elasticsearch cluster

Elastic Stack Elasticsearch
1

Hi, i'm trying to set up a cluster with 3 nodes

es-node1
es-node2
es-node3

installed elasticsearch 7.4 on 3 nodes con configure elasticsearch.yml with the following:

cluster.name: es-cluster
node.name: es-node1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 0.0.0.0
discovery.seed_hosts: ["es-node1"]
cluster.initial_master_nodes: ["es-node1"]

however when i start nodes each node starts its own cluster:

[root@es-node1 elasticsearch]# curl -XGET http://192.168.90.225:9200/_cluster/health?pretty
{
  "cluster_name" : "es-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 85,
  "active_shards" : 85,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 44,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 65.89147286821705
}

Selinux is in permissive and firewall down to avoid trouble.

Any idea?

1 Like
2

Finally made it work, had to start each node with a slightliy different configuration

node1:
cluster.name: es-cluster
node.name: es-node1
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.225
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1

node2: 
cluster.name: es-cluster
node.name: es-node2
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.226
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1
 - es-node2

node3:
cluster.name: es-cluster
node.name: es-node3
node.master: true
node.data: true
bootstrap.memory_lock: true
network.host: 192.168.90.227
#network.host: 0.0.0.0
discovery.seed_hosts:
 - es-node1
 - es-node2
 - es-node3
cluster.initial_master_nodes:
 - es-node1
 - es-node2
 - es-node3
3

For the benefit of future readers, the solution described above contains a misconfiguration which can result in data loss. From the docs:

WARNING: You must set cluster.initial_master_nodes to the same list of nodes on each node on which it is set in order to be sure that only a single cluster forms during bootstrapping...

This is not what you have done.

1 Like
4

Thanks David, but with all nodes in cluster.initial_master_nodes cluster did not start.
However in the way i managed to get it started now i see all nodes forming the cluster:

curl --cacert /etc/elasticsearch/certs/ca.crt -u elastic -XGET https://192.168.90.225:9200/_cat/nodes?v
Enter host password for user 'elastic':
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.90.226           14          21   0    0.00    0.01     0.05 dilm      -      es-node2
192.168.90.227            9          21   0    0.00    0.01     0.05 dilm      *      es-node3
192.168.90.225            8          21   0    0.00    0.01     0.05 dilm      -      es-node1

So i guess i got a single cluster, what else do you think might have been wrong?

thanks

5

Fortunately it looks like you did indeed get a single cluster, but this was luck. You could have ended up with two.

All three nodes is the correct setting for cluster.initial_master_nodes so I suspect something else was going wrong there. Do you have logs from the case where the cluster did not start? If so can you share them here?

6

Unfortunatelly i dont have logs, whenever it did not work i deleted the logs prior to some modification to avoid having to dig to much into the logs on the next try.

regards

7

Ok, not to worry. If you manage to reproduce it again then please get back in touch and we'll try and work out what's happening.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.