Elasticsearch: adding a second node to the cluster - [node-1] master not discovered yet: have discovered [{node-1}

Elastic Stack Elasticsearch
1

I am trying to obtain a two nodes cluster.
First node, master (master-1):

cluster.name: "admin_system_elk"
network.host: 0.0.0.0
node.name: "master-1"
discovery.seed_hosts: "localhost:9300"
cluster.initial_master_nodes: "master-1"
http.port: 9200
transport.port: 9300
node.data: true
node.master: true
#bootstrap.memory_lock: true

xpack.license.self_generated.type: basic
xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "http.p12"

Second node, not master (node-1):

cluster.initial_master_nodes: master-1
cluster.name: admin_system_elk
discovery.seed_hosts: [IP_OF_MASTER_1]:9300
http.port: 9200
node.data: true
node.master: false
node.name: node-1
transport.port: 9300
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: http.p12
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12
xpack.security.transport.ssl.verification_mode: certificate




#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):

path.data: /opt/elasticsearch/data

path.logs: /opt/elasticsearch/logs


action.auto_create_index: true

I am able to telnet [IP_OF_MASTER_1] 9300 without error.

Logs on node-1:

[2020-07-24T17:14:25,470][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet: have discovered [{node-1}{MB9wyKmvQPaeA1-GZte6Dg}{tdYj5WsBRdynk6_dzYqRxA}{127.0.0.1}{127.0.0.1:9300}{dil}{ml.machine_memory=33625956352, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [[IP_OF_MASTER_1] :9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0

This log is repeated every 10 seconds.

Logs on master-1:

{"type": "server", "timestamp": "2020-07-24T14:51:31,463Z", "level": "INFO", "component": "o.e.c.r.a.AllocationService", "cluster.name": "admin_system_elk", "node.name": "master-1", "message": "Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_task_manager_1][0], [.apm-agent-configuration][0], [.kibana_1][0]]]).", "cluster.uuid": "dvelqHyZQX69AhTzMaHs9Q", "node.id": "MVG8PHMwSZ6z1Re4OLsd9Q"  }

ELK 7.6.2

2

Could you try changing the line

discovery.seed_hosts: "localhost:9300"

the same as node.

discovery.seed_hosts: [IP_OF_MASTER_1]:9300
3

did not fix it

4

The second node also need network.host set to 0.0.0.0.

5

still getting

    [ node-1] master not discovered yet: have discovered [{node-1}{MB9wyKmvQPaeA1-GZte6Dg}{lpzawyqSQ5e6TTyQ5vSynA}{51.15.17.189}{51.15.17.189:9300}{dil}{ml.machine_memory=33625956352, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [MASTER_IP:9300] from hosts providers and [] from last-known cluster state;
            node term 0, last-accepted version 0 in term 0
6

master side seems okay:

{"type": "server", "timestamp": "2020-07-27T08:48:58,132Z", "level": "INFO", "component": "o.e.c.s.ClusterApplierService", "cluster.name": "admin_system_elk", "node.name": "master-1", "message": "master node changed {previous [], current [{master-1}{MVG8PHMwSZ6z1Re4OLsd9Q}{1syLdPA5TFyyAZDFpJmTDA}{172.29.0.2}{172.29.0.2:9300}{dilm}{ml.machine_memory=67328880640, xpack.installed=true, ml.max_open_jobs=20}]}, term: 102, version: 13424, reason: Publication{term=102, version=13424}" }

telnet localhost 9300 ->

Trying ::1...
Connected to localhost.
Escape character is '^]'.
7

master is running through docker, can there be special networking needed ?
it is not a firewall issue for sure, as I deactivated firewall on both sides and it did not work better

8

the issue was docker using docker local network ip (172.X.X.X) for network.publish.
adding network.publish: [IP OF MASTER] to the master elasticsearch.yml configuration fixed the discovery issue.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.