# Could not get cluster status after master node goes down **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2018, 9:55am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455 "2018-05-18T09:55:08Z")\ **Posts on this page:** 12\ **Page:** 3
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 7, 2018, 9:01am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/41 "2018-06-07T09:01:22Z")
``` at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:345) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.onResponse(TransportReplicationAction.java:270) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:924) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$1.onResponse(TransportReplicationAction.java:921) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.index.shard.IndexShardOperationsLock.acquire(IndexShardOperationsLock.java:151) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.index.shard.IndexShard.acquirePrimaryOperationLock(IndexShard.java:1659) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction.acquirePrimaryShardReference(TransportReplicationAction.java:933) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction.access$500(TransportReplicationAction.java:92) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncPrimaryAction.doRun(TransportReplicationAction.java:291) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:266) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryOperationTransportHandler.messageReceived(TransportReplicationAction.java:248) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:654) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] [2018-06-07T09:45:01,773][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [172.22.107.22:10000] failed to execute on node [b1rXMHeJTh2LfTN1klzHzA] org.elasticsearch.transport.NodeNotConnectedException: [172.22.107.21:10000][172.22.107.21:9300] Node not connected at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.7.jar:5.6.7] [2018-06-07T09:45:01,773][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [172.22.107.22:10000] failed to execute on node [b1rXMHeJTh2LfTN1klzHzA] org.elasticsearch.transport.NodeNotConnectedException: [172.22.107.21:10000][172.22.107.21:9300] Node not connected at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:640) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TcpTransport.getConnection(TcpTransport.java:117) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:540) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:516) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.start(TransportNodesAction.java:197) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:89) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.nodes.TransportNodesAction.doExecute(TransportNodesAction.java:52) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:170) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:84) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:408) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:730) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.nodesStats(AbstractClient.java:826) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.rest.action.admin.cluster.RestNodesStatsAction.lambda$prepareRequest$13(RestNodesStatsAction.java:189) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:262) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:200) [elasticsearch-5.6.7.jar:5.6.7] ``` all the posted logs belong to new master node (172.22.107.22). PS. I think it would be nice to have the possiblilty to attach the .txt files to the posts. Sometimes it's not easy to share the files... ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 7, 2018, 9:07am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/42 "2018-06-07T09:07:48Z")
The following lines are the logs for the node 172.22.107.20: ``` [2018-06-07T09:42:29,407][INFO][o.e.n.Node] [172.22.107.20:10000] initializing ... [2018-06-07T09:42:29,556][INFO][o.e.e.NodeEnvironment] [172.22.107.20:10000] using [1] data paths, mounts [[/opt (/dev/sdb)]], net usable_space [112.7gb], net total_space [150gb], spins? [possibly], types [btrfs] [2018-06-07T09:42:29,556][INFO][o.e.e.NodeEnvironment] [172.22.107.20:10000] heap size [11.9gb], compressed ordinary object pointers [true] [2018-06-07T09:42:29,692][INFO][o.e.n.Node] [172.22.107.20:10000] node name [172.22.107.20:10000], node ID [w49SCPmcRvSsW0_948B_rA] [2018-06-07T09:42:29,693][INFO][o.e.n.Node] [172.22.107.20:10000] version[5.6.7], pid[24151], build[4669214/2018-01-25T21:14:50.776Z], OS[Linux/3.10.0-693.21.1.el7.x86_64/amd64], JVM[Oracle Corp oration/Java HotSpot(TM) 64-Bit Server VM/1.8.0_162/25.162-b12] [2018-06-07T09:42:29,693][INFO][o.e.n.Node] [172.22.107.20:10000] JVM arguments [-Xms12g, -Xmx12g, -Xss1m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOc cupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKey SetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/opt/bdm _current//log/es_heapdump_20180607-0942.hprof, -Dlog4j2.disable.jmx=true, -Delasticsearch, -Des.pidfile=/opt/bdm_current//pid/172.22.107.20:10000/elasticsearch.pid, -Des.path.home=/opt/bdm_current/, -Des.path.l ogs=/opt/bdm_current//log, -Des.path.data=/opt/bdm4_data, -Des.path.work=/tmp/elasticsearch/172.22.107.20:10000] [2018-06-07T09:42:30,982][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [aggs-matrix-stats] [2018-06-07T09:42:30,982][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [ingest-common] [2018-06-07T09:42:30,982][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [lang-expression] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [lang-groovy] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [lang-mustache] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [lang-painless] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [parent-join] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [percolator] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [reindex] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [transport-netty3] [2018-06-07T09:42:30,983][INFO][o.e.p.PluginsService] [172.22.107.20:10000] loaded module [transport-netty4] [2018-06-07T09:42:30,984][INFO][o.e.p.PluginsService] [172.22.107.20:10000] no plugins loaded [2018-06-07T09:42:33,220][INFO][o.e.d.DiscoveryModule] [172.22.107.20:10000] using discovery type [zen] [2018-06-07T09:42:34,321][INFO][o.e.n.Node] [172.22.107.20:10000] initialized [2018-06-07T09:42:34,322][INFO][o.e.n.Node] [172.22.107.20:10000] starting ... [2018-06-07T09:42:34,529][INFO][o.e.t.TransportService] [172.22.107.20:10000] publish_address {172.22.107.20:9300}, bound_addresses {[::]:9300} [2018-06-07T09:42:34,541][INFO][o.e.b.BootstrapChecks] [172.22.107.20:10000] bound or publishing to a non-loopback address, enforcing bootstrap checks [2018-06-07T09:42:37,849][INFO][o.e.c.s.ClusterService] [172.22.107.20:10000] detected_master {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}, added {{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300},{172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300},}, reason: zen-disco-receive(from master [master {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300} committed version [45]]) [2018-06-07T09:42:39,887][INFO][o.e.h.n.Netty4HttpServerTransport] [172.22.107.20:10000] publish_address {172.22.107.20:9200}, bound_addresses {[::]:9200} [2018-06-07T09:42:39,887][INFO][o.e.n.Node] [172.22.107.20:10000] started [2018-06-07T09:44:57,366][INFO][o.e.d.z.ZenDiscovery] [172.22.107.20:10000] master_left [{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}], reason [shut_down] [2018-06-07T09:44:57,369][WARN][o.e.d.z.ZenDiscovery] [172.22.107.20:10000] master left (reason = shut_down), current nodes: nodes: {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300}, master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300} {172.22.107.20:10000}{w49SCPmcRvSsW0_948B_rA}{VnlkmgbtQuO793YmrfYDnA}{172.22.107.20}{172.22.107.20:9300}, local ``` ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 7, 2018, 9:08am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/43 "2018-06-07T09:08:15Z")
``` [2018-06-07T09:44:58,036][DEBUG][o.e.a.a.c.s.TransportClusterStateAction] [172.22.107.20:10000] no known master node, scheduling a retry [2018-06-07T09:44:58,133][DEBUG][o.e.a.a.c.h.TransportClusterHealthAction] [172.22.107.20:10000] no known master node, scheduling a retry [2018-06-07T09:45:00,448][INFO][o.e.c.s.ClusterService] [172.22.107.20:10000] detected_master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300}, reason: zen-disco-receive(from master [master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300} committed version [76]]) [2018-06-07T09:45:00,458][WARN][o.e.c.NodeConnectionsService] [172.22.107.20:10000] failed to connect to node {172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300} (tried [1] times) org.elasticsearch.transport.ConnectTransportException: [172.22.107.21:10000][172.22.107.21:9300] connect_timeout[30s] at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:363) ~[?:?] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:570) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:473) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:342) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:107) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.7.jar:5.6.7] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.7.jar:5.6.7] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 172.22.107.21/172.22.107.21:9300 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?] ... 1 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?] at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:352) ~[?:?] at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) ~[?:?] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) ~[?:?] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) ~[?:?] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?] ... 1 more [2018-06-07T09:45:00,828][INFO][o.e.c.s.ClusterService] [172.22.107.20:10000] removed {{172.22.107.21:10000}{b1rXMHeJTh2LfTN1klzHzA}{00vO7HZuQcq2LfLYZ-zQAg}{172.22.107.21}{172.22.107.21:9300},}, reason: zen-disco-receive(from master [master {172.22.107.22:10000}{ds3DRQbwR2qQg7S9x-ljfw}{CTegKdEiQh2i2olrKdibrg}{172.22.107.22}{172.22.107.22:9300} committed version [77]]) (END) ``` ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 7, 2018, 9:12am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/44 "2018-06-07T09:12:55Z")
Upload to [gist.github.com](http://gist.github.com) if you wish. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 7, 2018, 9:44am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/45 "2018-06-07T09:44:52Z")
On which machine did you run then: ```auto curl -XGET localhost:9200/_cluster/health?pretty ``` ? ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 7, 2018, 9:48am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/46 "2018-06-07T09:48:38Z")
On both. I'm also using head plugin which is connected to 172.22.107.22 using port forwarding. After stopping the master it doesn't respond as well. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 7, 2018, 9:57am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/47 "2018-06-07T09:57:38Z")
I believe you have something wrong in your network configuration. You really need to check your firewall rules. This kind of problem reported on this forum is caused by this 99% of the time. ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 11, 2018, 12:40pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/48 "2018-06-11T12:40:10Z")
After many tests I found that this problem doesn't occur if I start elasticsearch manually with the java command and same arguments. So it could not be a problem with network or hardware, IMHO. We have a java application which read the jvm configuration from a xml file and start ES process using ``` ProcessBuilder pb = new ProcessBuilder(command); pb = pb.redirectErrorStream(true); pb.start() ``` It seems that something goes wrong when a nodes crashes. I've started ES instances manually using the following command (not using our java application to start it) and tested the cluster HA and everything was just fine : > sudo -u bdm java -Xms12g -Xmx12g -Xss1m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/bdm\_current//log/es\_heapdump\_20180608-0741.hprof -Dlog4j2.disable.jmx=true -Delasticsearch -Des.pidfile=/opt/bdm\_current//pid/172.22.107.20:10000/elasticsearch.pid -Des.path.home=/opt/bdm\_current/ -Des.path.logs=/opt/bdm\_current//log -Des.path.data=/opt/bdm4\_data -Des.path.work=/tmp/elasticsearch/172.22.107.20:10000 -cp /opt/bdm\_current//es-lib/\*:/opt/bdm\_current//modules org.elasticsearch.bootstrap.Elasticsearch these command copied from the logs in parent java process, so the same parameters will be used to start ES. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 11, 2018, 12:55pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/49 "2018-06-11T12:55:13Z")
We are not supporting launching elasticsearch from another process. I don't know what you can do here. But why are you doing this and not starting elasticsearch as you would start MySQL for example? ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 11, 2018, 1:00pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/50 "2018-06-11T13:00:37Z")
we wanted to simplify the work of our customers not to deal with many configuration files. It is easier for them to have all the configurations of the application(s) in a single file.... ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [June 15, 2018, 7:33am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/51 "2018-06-15T07:33:17Z")
Hi David, I've changed the way of starting ES by using the scripts, however the ES start script get executed by the parent script which start all the processes, so no extra works for the system admins as well. After that HA working just fine. Thank you for your great advices. Cheers, Vahid ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 13, 2018, 7:33am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/52 "2018-07-13T07:33:20Z")
Closed on July 13, 2018, 7:33am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455.md?page=2)