Hi, i have configured Elasticsearch through docker swarm. When i do deploy of swarm every container in nodes are up and running but when i checked elastic search from browser getting master not discovered exception. Help me in trouble shooting.
[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
Thanks in advance.
Please don't post screenshots of text, they're impossible to search and are unreadable for those of us using screen readers. Instead include them in your post, using the </> button to format them properly.
The first troubleshooting step is to look at the Elasticsearch logs.
1 Like
Hi David, Find the logs with the master and data nodes configuration.
<", "cluster.name": "elastic", "node.name": "master", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [["master", "data1", "data2"]] to bootstrap a cluster: have discovered [{master}{zCahhFrBT9K4zX3mvUtmGQ}{C65UJBdbRne48YnZQBvMJQ}{10.0.5.3}{10.0.5.3:9300}{ilm}{ml.machine_memory=4127330304, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using from hosts providers and [{master}{zCahhFrBT9K4zX3mvUtmGQ}{C65UJBdbRne48YnZQBvMJQ}{10.0.5.3}{10.0.5.3:9300}{ilm}{ml.machine_memory=4127330304, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }
KFCES_master.1.888qop7bmeva@master | {"type": "server", "timestamp": "2020-11-21T10:55:06,487Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elastic", "node.name": "master", "message": "failed to resolve host [["master"]",
KFCES_master.1.888qop7bmeva@master | "stacktrace": ["java.lang.IllegalArgumentException: Invalid bracketed host/port range: ["master"",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:500) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:803) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
KFCES_master.1.888qop7bmeva@master | {"type": "server", "timestamp": "2020-11-21T10:55:06,488Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elastic", "node.name": "master", "message": "failed to resolve host ["data1"]",
KFCES_master.1.888qop7bmeva@master | "stacktrace": ["java.net.UnknownHostException: "data1"",
KFCES_master.1.888qop7bmeva@master | "at java.net.InetAddress$CachedAddresses.get(InetAddress.java:798) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.net.InetAddress.getAllByName0(InetAddress.java:1489) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.net.InetAddress.getAllByName(InetAddress.java:1348) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.net.InetAddress.getAllByName(InetAddress.java:1282) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TcpTransport.parse(TcpTransport.java:528) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TcpTransport.addressesFromString(TcpTransport.java:470) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.transport.TransportService.addressesFromString(TransportService.java:803) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.discovery.SeedHostsResolver.lambda$resolveHostsLists$0(SeedHostsResolver.java:144) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]",
KFCES_master.1.888qop7bmeva@master | "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703) ~[elasticsearch-7.5.2.jar:7.5.2]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
KFCES_master.1.888qop7bmeva@master | "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
KFCES_master.1.888qop7bmeva@master | {"type": "server", "timestamp": "2020-11-21T10:55:06,488Z", "level": "WARN", "component": "o.e.d.SeedHostsResolver", "cluster.name": "elastic", "node.name": "master", "message": "timed out after [5s] resolving host ["data2"]]" }>
Master node configuration:
master:
image: kfckibana.azurecr.io/elasticsearch:7.5.2
environment:
- cluster.name=elastic
- node.name=master
- "ES_JAVA_OPTS=-Xms1962m -Xmx1962m"
- node.master=true
- node.data=false
- discovery.zen.ping.unicast.hosts=["master","data1","data2"]
- discovery.zen.minimum_master_nodes=1
- bootstrap.memory_lock=false
- cluster.initial_master_nodes=["master","data1","data2"]
- ELASTIC_PASSWORD=administrator098
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certificates/master/master.key
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certificates/master/master.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certificates/master/master.crt
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certificates/master/master.key
- network.publish_host:master
deploy:
placement:
constraints:
- node.labels.es-node == 1
restart: always
ports:
- target: 9200
published: 9200
protocol: tcp
mode: host
logging:
driver: "json-file"
options:
max-size: "5000k"
max-file: "10"
volumes:
- es_data_store:/usr/share/elasticsearch/data
- certs:/usr/share/elasticsearch/config/certificates
Data1 node:
data1:
image: kfckibana.azurecr.io/elasticsearch:7.5.2
environment:
- cluster.name=elastic
- node.name=data1
- "ES_JAVA_OPTS=-Xms1962m -Xmx1962m"
- node.master=false
- node.data=true
- discovery.zen.ping.unicast.hosts=["master","data1","data2"]
- discovery.zen.minimum_master_nodes=1
- bootstrap.memory_lock=false
- cluster.initial_master_nodes=["master","data1","data2"]
- ELASTIC_PASSWORD=administrator098
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certificates/data1/data1.key
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certificates/data1/data1.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certificates/data1/data1.crt
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certificates/data1/data1.key
- network.publish_host:data1
deploy:
placement:
constraints:
- node.labels.es-node == 2
ports:
- target: 9200
published: 9200
protocol: tcp
mode: host
restart: always
logging:
driver: "json-file"
options:
max-size: "5000k"
max-file: "10"
volumes:
- es_data_store:/usr/share/elasticsearch/data
- certs:/usr/share/elasticsearch/config/certificates
Data2 configurtaion:
image: kfckibana.azurecr.io/elasticsearch:7.5.2
environment:
- cluster.name=elastic
- node.name=data2
- "ES_JAVA_OPTS=-Xms1962m -Xmx1962m"
- node.master=false
- node.data=true
- discovery.zen.ping.unicast.hosts=["master","data1","data2"]
- discovery.zen.minimum_master_nodes=1
- bootstrap.memory_lock=false
- cluster.initial_master_nodes=["master","data1","data2"]
- ELASTIC_PASSWORD=administrator098
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=/usr/share/elasticsearch/config/certificates/data2/data2.key
- xpack.security.http.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.http.ssl.certificate=/usr/share/elasticsearch/config/certificates/data2/data2.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certificates/ca/ca.crt
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certificates/data2/data2.crt
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certificates/data2/data2.key
- network.publish_host:data2
deploy:
placement:
constraints:
- node.labels.es-node == 3
ports:
- target: 9200
published: 9200
protocol: tcp
mode: host
restart: always
logging:
driver: "json-file"
options:
max-size: "5000k"
max-file: "10"
volumes:
- es_data_store:/usr/share/elasticsearch/data
- certs:/usr/share/elasticsearch/config/certificates
In which place i am doing error. why i am getting error like master is not discoverd.
That would indicate a DNS issue.
Can you ping master to see if it resolves?
kvantum@master:~$ ping master
PING master(master (2400:8904::f03c:92ff:fed9:76aa)) 56 data bytes
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=2 ttl=64 time=0.038 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=3 ttl=64 time=0.036 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=4 ttl=64 time=0.038 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=5 ttl=64 time=0.038 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=6 ttl=64 time=0.037 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=7 ttl=64 time=0.037 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=8 ttl=64 time=0.051 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=9 ttl=64 time=0.039 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=10 ttl=64 time=0.037 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=11 ttl=64 time=0.038 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=12 ttl=64 time=0.032 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=13 ttl=64 time=0.037 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=14 ttl=64 time=0.039 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=15 ttl=64 time=0.045 ms
64 bytes from master (2400:8904::f03c:92ff:fed9:76aa): icmp_seq=16 ttl=64 time=0.039 ms
I am able to ping all thenodes .and if do telnet from master to data using 9200 port,it is also connected.
The issue is here:
These nodes aren't all master nodes.
2 Likes
Hi have changed data nodes to master nodes as below.
environment:
- cluster.name=elastic
- node.name=data2
- "ES_JAVA_OPTS=-Xms1962m -Xmx1962m"
- node.master=true
- node.data=true
- discovery.zen.ping.unicast.hosts=["master","data1","data2"]
- discovery.zen.minimum_master_nodes=1
- bootstrap.memory_lock=false
- cluster.initial_master_nodes=["master","data1","data2"]
now also same issue i am facing
I am able to open 3 nodes from browser ,its working but it is not able to take master.Could you provide any troubleshoot steps.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.