i'm trying to build up the 3-master 3-nodes elastic search stack in azure using vm.
when i configure the elasticsearch with azure cloud discovery plugin i dont see that both master can communicate & it is not able to join the cluster.
Elasticsearch version: 6.7.2
os: ubuntu 16.04 LTS
what im trying to solve is,
i wanted elasticsearch to discover new nodes and join to the cluster automatically.
Please suggest how i can reslove this issue and implement node autodiscovery?
my current elastic search.yml looks like below:
cluster.name: "esprod"
node.name: "master-3"
path.logs: /var/log/elasticsearch
path.data: /var/lib/elasticsearch
discovery.zen.minimum_master_nodes: 2
node.master: true
node.data: false
network.host: 0.0.0.0
cloud:
azure:
management:
subscription.id: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
cloud.service.name: esprod
keystore:
path: /etc/elasticsearch/azurekeystore.pkcs12
password: xxxxxxxxx
discovery:
zen.hosts_provider: azure
azure:
host:
type: private_ip
endpoint:
name: elasticsearch
i tested the pkcs12 which is fine.
Logs:
master-3:
[2019-09-25T17:55:04,722][WARN ][o.e.d.z.ZenDiscovery ] [master-3] not enough master nodes discovered during pinging (found [[Candidate{node={master-3}{Fpo_LFhORbShL26UAUrwdw}{ZDI1vbyNRnGCJHgqUl5xPA}{172.16.0.5}{172.16.0.5:9300}{ml.machine_memory=8341356544, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-09-25T17:57:08,006][INFO ][o.a.h.i.e.RetryExec ] [master-3] I/O exception (java.net.SocketException) caught when processing request to {s}->https://management.core.windows.net:443: Connection reset
[2019-09-25T17:57:08,007][INFO ][o.a.h.i.e.RetryExec ] [master-3] Retrying request to {s}->https://management.core.windows.net:443
[2019-09-25T17:57:08,350][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-3] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T17:57:41,644][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-3] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T17:58:15,042][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-3] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T17:58:48,523][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-3] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T17:59:21,855][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-3] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
master-2:
2019-09-25T19:08:24,332][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-2] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T19:08:27,338][INFO ][o.e.d.z.ZenDiscovery ] [master-2] failed to send join request to master [{master-3}{Fpo_LFhORbShL26UAUrwdw}{ZDI1vbyNRnGCJHgqUl5xPA}{172.16.0.5}{172.16.0.5:9300}{ml.machine_memory=8341356544, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[master-3][172.16.0.5:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[master-2][127.0.0.1:9300] handshake failed. unexpected remote node {master-3}{Fpo_LFhORbShL26UAUrwdw}{ZDI1vbyNRnGCJHgqUl5xPA}{172.16.0.5}{172.16.0.5:9300}{ml.machine_memory=8341356544, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; ]
[2019-09-25T19:08:27,635][WARN ][o.e.d.a.c.AzureUnicastHostsProvider] [master-2] can not get list of azure nodes: [can not get list of azure nodes]. Returning empty list of nodes.
[2019-09-25T19:08:30,640][INFO ][o.e.d.z.ZenDiscovery ] [master-2] failed to send join request to master [{master-3}{Fpo_LFhORbShL26UAUrwdw}{ZDI1vbyNRnGCJHgqUl5xPA}{172.16.0.5}{172.16.0.5:9300}{ml.machine_memory=8341356544, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}], reason [RemoteTransportException[[master-3][172.16.0.5:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[master-2][127.0.0.1:9300] handshake failed. unexpected remote node {master-3}{Fpo_LFhORbShL26UAUrwdw}{ZDI1vbyNRnGCJHgqUl5xPA}{172.16.0.5}{172.16.0.5:9300}{ml.machine_memory=8341356544, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}];