Nodes didn't discovery in ElasticSearch Cluster

SilasMuniz1 · July 13, 2022, 5:00pm

Hello,

I am setting up one cluster with two nodes to elasticsearch. I am using virtualbox with ubuntu 22.04 image.

I already configurated all nodes with these elasticsearch.yml:

Node1:

root@osboxes:/var/log/elasticsearch# cat /etc/elasticsearch/elasticsearch.yml

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

Elasticsearch Guide | Elastic

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-cluster

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node01

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.memory_lock: false

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

By default Elasticsearch is only accessible on localhost. Set a different

address here to expose this node on the network:

network.host: 192.168.0.24

By default Elasticsearch listens for HTTP traffic on the first free port it

finds starting at 9200. Set a specific HTTP port here:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["192.168.0.24","192.168.0.25"]

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes: ["node01", "node02"]

For more information, consult the discovery and cluster formation module documentation.

--------------------------------- Readiness ----------------------------------

Enable an unauthenticated TCP readiness endpoint on localhost

#readiness.port: 9399

---------------------------------- Various -----------------------------------

Allow wildcard deletion of indices:

#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

The following settings, TLS certificates, and keys have been automatically

generated to configure Elasticsearch security features on 12-07-2022 20:10:54

--------------------------------------------------------------------------------

Enable security features

xpack.security.enabled: false

xpack.security.enrollment.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

Create a new cluster with the current node only

Additional nodes can still join the cluster later

#cluster.initial_master_nodes: ["osboxes"]

Allow HTTP API connections from anywhere

Connections are encrypted and require user authentication

http.host: 0.0.0.0

Allow other nodes to join the cluster from anywhere

Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

Node2:

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

Elasticsearch Guide | Elastic

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: my-cluster

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: node02

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

bootstrap.memory_lock: false

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

By default Elasticsearch is only accessible on localhost. Set a different

address here to expose this node on the network:

network.host: 192.168.0.25

By default Elasticsearch listens for HTTP traffic on the first free port it

finds starting at 9200. Set a specific HTTP port here:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["192.168.0.24","192.168.0.25"]

Bootstrap the cluster using an initial set of master-eligible nodes:

cluster.initial_master_nodes: ["node01", "node02"]

For more information, consult the discovery and cluster formation module documentation.

--------------------------------- Readiness ----------------------------------

Enable an unauthenticated TCP readiness endpoint on localhost

#readiness.port: 9399

---------------------------------- Various -----------------------------------

Allow wildcard deletion of indices:

#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

The following settings, TLS certificates, and keys have been automatically

generated to configure Elasticsearch security features on 13-07-2022 01:46:47

--------------------------------------------------------------------------------

Enable security features

xpack.security.enabled: false

xpack.security.enrollment.enabled: true

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

Create a new cluster with the current node only

Additional nodes can still join the cluster later

#cluster.initial_master_nodes: ["silas-VirtualBox"]

Allow HTTP API connections from anywhere

Connections are encrypted and require user authentication

http.host: 0.0.0.0

Allow other nodes to join the cluster from anywhere

Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

When second node is starting I didn't see any information about /var/log/elasticsearch/my-cluster.log.

Both Elasticsearch each node are up, but each one with the same cluster name but different cluster id:

Node1:

root@osboxes:/var/log/elasticsearch# curl -H "Content-Type: application/json" -XGET 127.0.0.1:9200/_cluster/state/master?pretty
{
"cluster_name" : "my-cluster",
"cluster_uuid" : "AetnBE4HSHWZBxV104hYsQ"
}

Node2:

root@silas-VirtualBox:/var/log/elasticsearch# curl -H "Content-Type: application/json" -XGET 127.0.0.1:9200/_cluster/state/master?pretty
{
"cluster_name" : "my-cluster",
"cluster_uuid" : "7sH6JOYoT-iCgTubuPXgpw"
}

Could anyone help me?

elasticforme · July 13, 2022, 5:15pm

you have two seperate cluster with one node each.
here is how you do it.

stop both cluster.
remove /var/lib/elasticsearch/* dir ( as it now has data and they think they are seperate cluster)
change cluster.initial_master_nodes:["node01"]
number 3 should be same on both node
start node 1 first. make sure it is up and running
start node2 and it will join to cluster that node1 just created.
now you can stop both and completly remove this "cluster.initial_master_nodes or comment out.
start both in any sequence and it should work.

SilasMuniz1 · July 13, 2022, 5:18pm

I have 2 VMs each one with elasticsearch running. I want join these nodes to form a cluster.

elasticforme · July 13, 2022, 5:22pm

then do what I said above.
currently they think they are separate cluster. you got a redo process

SilasMuniz1 · July 14, 2022, 12:29pm

Sachin,

It worked !!!

silas@ELK-node2:~$ curl -H "Content-Type: application/json" -XGET 127.0.0.1:9200/_cluster/health?pretty
{
  "cluster_name" : "BancoBrasil",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 10,
  "active_shards" : 20,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
silas@ELK-node2:~$```


Thanks a lot !!!

Christian_Dahlqvist · July 14, 2022, 12:36pm

If you are trying to set up a highly available cluster - be aware that you need a minimum of 3 nodes in the cluster to get high availability and the ability for the cluster to continue operating even if you lose one node. See the documentation for further details.

system · August 11, 2022, 12:37pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cluster cannot find nodes Elasticsearch	3	287	September 7, 2021
Talking nodes on same machine Elasticsearch	2	421	September 7, 2018
Elasticsearch 7.11 with 3 nodes Elasticsearch	7	773	April 9, 2021
The cluster is not working Elasticsearch	7	207	May 28, 2024
Configuration for cluster.initial_master_nodes Elasticsearch	11	1534	May 23, 2020