# Client node unable to discover master node after master restarts and gets new IP **URL:** \ **Category:** Elasticsearch\ **Created:** [July 19, 2018, 12:05am UTC](https://discuss.elastic.co/t/client-node-unable-to-discover-master-node-after-master-restarts-and-gets-new-ip/140635 "2018-07-19T00:05:39Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Sonia\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sonia_gupta/32/49646_2.png) [@Sonia\_Gupta](https://discuss.elastic.co/u/Sonia_Gupta)\ **Post date:** [July 19, 2018, 12:05am UTC](https://discuss.elastic.co/t/client-node-unable-to-discover-master-node-after-master-restarts-and-gets-new-ip/140635/1 "2018-07-19T00:05:40Z")
I have a 1 client/ 2 data/ 3 master Elasticsearch 6.2.2 cluster spread over 3 node vpshere environment. This is a kubernetes system. Data and master are statefulsets. Master has a service sitting on it acting as the discovery service. The installation is successful and we can see the data in Kibana, however, If all 3 masters happen to go down together, they come up with new IPs and discover each other, but data and client cant seem to discover the new masters, the logs tell me they are still trying to hit the old IPs. Master log: `[2018-07-18T23:13:52,533][INFO][o.e.b.BootstrapChecks] [platform-elasticsearch-master-0] bound or publishing to a non-loopback address, enforcing bootstrap checks [2018-07-18T23:13:52,943][INFO][o.e.m.j.JvmGcMonitorService] [platform-elasticsearch-master-0] [gc][1] overhead, spent [372ms] collecting in the last [1s] [2018-07-18T23:13:55,890][INFO][o.e.c.s.ClusterApplierService] [platform-elasticsearch-master-0] detected_master {platform-elasticsearch-master-1}{Or4qtPKjRyajMI-iC9mnwg}{Aw3yHwDhSOu3DiocgUParg}{172.24.0.99}{172.24.0.99:9300}, added {{platform-elasticsearch-client-78d74649fc-7976p}{TkuEXJttR1C93sOuYheVTg}{6vsOtJxFTC6AdaPzNrJNbQ}{172.24.2.212}{172.24.2.212:9300},{platform-elasticsearch-master-2}{bpIqzNyKS86Sk8vPRHSy7A}{RD3XUV1TTKaD1cjOSuxf7g}{172.24.1.104}{172.24.1.104:9300},{platform-elasticsearch-data-0}{cpsl1MgFT0-c15IqgVyX7w}{hVWwSzwuQ6arlVVDEO_JkQ}{172.24.2.214}{172.24.2.214:9300},{platform-elasticsearch-data-1}{CKsi0wfrQtC421UuhS4EUQ}{tn-w2myFQkmN1nV1kaHB0g}{172.24.1.105}{172.24.1.105:9300},{platform-elasticsearch-master-1}{Or4qtPKjRyajMI-iC9mnwg}{Aw3yHwDhSOu3DiocgUParg}{172.24.0.99}{172.24.0.99:9300},}, reason: apply cluster state (from master [master {platform-elasticsearch-master-1}{Or4qtPKjRyajMI-iC9mnwg}{Aw3yHwDhSOu3DiocgUParg}{172.24.0.99}{172.24.0.99:9300} committed version [2]]) [2018-07-18T23:13:56,205][INFO][o.e.n.Node] [platform-elasticsearch-master-0] started` Client logs: `[2018-07-18T22:49:34,640][WARN][o.e.d.z.ZenDiscovery] [platform-elasticsearch-client-5d5cff75d9-k29x7] not enough master nodes discovered during pinging (found [[]], but needed [2]), pinging again [2018-07-18T22:49:35,635][WARN][o.e.c.NodeConnectionsService] [platform-elasticsearch-client-5d5cff75d9-k29x7] failed to connect to node {platform-elasticsearch-master-1}{IPe2h_kETombpzD2ZSnetA}{ACKXkGpVQ8CMv5td3tAHxQ}{172.24.1.101}{172.24.1.101:9300} (tried [7] times) org.elasticsearch.transport.ConnectTransportException: [platform-elasticsearch-master-1][172.24.1.101:9300] connect_timeout[30s] at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:163) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:616) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]` master/client config: cluster.name: ${CLUSTER\_NAME:true} node.name: ${NODE\_NAME:} node.master: ${NODE\_MASTER:} node.data: ${NODE\_DATA:} node.ingest: ${NODE\_INGEST:} network.host: ${NETWORK\_HOST:0.0.0.0} path: data: /usr/share/elasticsearch/data logs: /usr/share/elasticsearch/logs bootstrap: memory\_lock: ${MEMORY\_LOCK:false} http: enabled: ${HTTP\_ENABLE:false} discovery: zen: ping.unicast.hosts: ${DISCOVERY\_SERVICE:} minimum\_master\_nodes: ${MINIMUM\_NUMBER\_OF\_MASTERS:1} commit\_timeout: 60s publish\_timeout : 60s gateway.expected\_nodes: 5 gateway.expected\_master\_nodes: 3 gateway.expected\_data\_nodes: 2 gateway.recover\_after\_nodes: 2 gateway.recover\_after\_master\_nodes: 2 gateway.recover\_after\_data\_nodes: 1 NOTE: Restarting the client fixes the issue. But why is this happening? Please help! ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [July 19, 2018, 3:08am UTC](https://discuss.elastic.co/t/client-node-unable-to-discover-master-node-after-master-restarts-and-gets-new-ip/140635/2 "2018-07-19T03:08:17Z")
discovery.zen.ping.unicast.hosts: need add the new ip . ---
**Author:** ![Sonia\_Gupta](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sonia_gupta/32/49646_2.png) [@Sonia\_Gupta](https://discuss.elastic.co/u/Sonia_Gupta)\ **Post date:** [July 19, 2018, 3:54am UTC](https://discuss.elastic.co/t/client-node-unable-to-discover-master-node-after-master-restarts-and-gets-new-ip/140635/3 "2018-07-19T03:54:26Z")
I have a service that returns the IPs of all masters, Client is able to get them successfully during startup, but not again when the masters change the IPs, when I ping the service, I get new IPs ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2018, 3:54am UTC](https://discuss.elastic.co/t/client-node-unable-to-discover-master-node-after-master-restarts-and-gets-new-ip/140635/4 "2018-08-16T03:54:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.