# Could not get cluster status after master node goes down **URL:** \ **Category:** Elasticsearch\ **Created:** [May 18, 2018, 9:55am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455 "2018-05-18T09:55:08Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 9:55am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/1 "2018-05-18T09:55:08Z")
Hi, recently we have upgraded ES to 5.6.7 I have a two nodes cluster. After stopping master node, the command "curl -X GET localhost:9211/\_cluster/health doesn't respond. I've run jstack and seen that many threads are blocked. For example: Thread 22447: (state = BLOCKED) - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise) - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame) - java.util.concurrent.LinkedTransferQueue.awaitMatch(java.util.concurrent.LinkedTransferQueue$Node, java.util.concurrent.LinkedTransferQueue$Node, java.lang.Object, boolean, long) @bci=184, line=737 (Compiled frame) - java.util.concurrent.LinkedTransferQueue.xfer(java.lang.Object, boolean, int, long) @bci=286, line=647 (Compiled frame) - java.util.concurrent.LinkedTransferQueue.take() @bci=5, line=1269 (Compiled frame) - org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take() @bci=4, line=161 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=149, line=1067 (Compiled frame) - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1127 (Interpreted frame) - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=745 (Interpreted frame) or Thread 21494: (state = BLOCKED) - org.apache.logging.log4j.core.layout.TextEncoderHelper.writeChunkedEncodedText(java.nio.charset.CharsetEncoder, java.nio.CharBuffer, org.apache.logging.log4j.core.layout.ByteBufferDestination, java.nio.ByteB uffer, java.nio.charset.CoderResult) @bci=5, line=112 (Interpreted frame) - org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(java.nio.charset.CharsetEncoder, java.nio.CharBuffer, java.nio.ByteBuffer, org.apache.logging.log4j.core.layout.ByteBufferDestination, java.nio.charset.CoderResult) @bci=14, line=79 (Interpreted frame) - org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeChunkedText(java.nio.charset.CharsetEncoder, java.nio.CharBuffer, java.nio.ByteBuffer, java.lang.StringBuilder, org.apache.logging.log4j.core.layo ut.ByteBufferDestination) @bci=91, line=143 (Interpreted frame) - org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(java.nio.charset.CharsetEncoder, java.nio.CharBuffer, java.nio.ByteBuffer, java.lang.StringBuilder, org.apache.logging.log4j.core.layout.Byte BufferDestination) @bci=22, line=58 (Interpreted frame) - org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(java.lang.StringBuilder, org.apache.logging.log4j.core.layout.ByteBufferDestination) @bci=37, line=68 (Interpreted frame) - org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(java.lang.Object, org.apache.logging.log4j.core.layout.ByteBufferDestination) @bci=6, line=32 (Interpreted frame) After starting the master node again I see the following error messages in the logs: ![es-log](https://us1.discourse-cdn.com/elastic/original/3X/4/6/467752888c50a0718a0aef594a79d502bb75b5ce.png) Cheers, Vahid ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [May 18, 2018, 11:10am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/2 "2018-05-18T11:10:29Z")
Looks like a split brain situation to me. You may have a look at [this](https://www.elastic.co/guide/en/elasticsearch/guide/1.x/_important_configuration_changes.html) ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 12:09pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/3 "2018-05-18T12:09:15Z")
it was working fine with two nodes with version 1.7 . There are one replica shard for indices. So if one nodes goes off, cluster should work properly. And if the off node come up it should join as well.... ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 18, 2018, 12:11pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/4 "2018-05-18T12:11:06Z")
so you are just testing if one node goes down that you can still query the cluster. can you show us your elasticsearch.yml ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 12:15pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/5 "2018-05-18T12:15:42Z")
discovery.zen.ping.unicast.hosts: ["ip-first-node:9311","ip-second-node:9311"] cluster.name: cluster-name indices.ttl.interval: 86400s http.port: 9211 reindex.remote.whitelist: localhost:\* transport.tcp.compress: true transport.tcp.port: 9311 path.repo: backup-folder discovery.zen.minimum\_master\_nodes: 1 bootstrap.system\_call\_filter: false path.data: data-folder network.host: 0.0.0.0 node.name: ip-first-node:10110 action.auto\_create\_index: false ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 18, 2018, 12:17pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/6 "2018-05-18T12:17:30Z")
> [@Vahid](#): > > discovery.zen.minimum\_master\_nodes: 1 this should be `2`. The quorum is as per here: [Important Elasticsearch configuration | Elasticsearch Reference [5.6] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/important-settings.html#minimum_master_nodes) ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 12:18pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/7 "2018-05-18T12:18:27Z")
So if I have only 2 nodes, and one crash, with "discovery.zen.minimum\_master\_nodes: 2" still works? Actually this problem reported by our customers and they have 3 nodes, with the minimum\_master\_nodes set on 2. So I don't think this is the root cause. ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 18, 2018, 12:23pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/8 "2018-05-18T12:23:00Z")
Then your repro on a two node cluster with quorum set to 1 is wrong. please revise inline with what your customer sees and report back. ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 12:40pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/9 "2018-05-18T12:40:39Z")
So you mean that we could not have a cluster with two nodes, if one crash the another node can not service anymore? These are additional configurations which are applied on a three nodes cluster: cluster.routing.allocation.awareness.attributes: sitename node.master: true node.data: true discovery.zen.fd.ping\_timeout: 30s discovery.zen.fd.ping\_retries: 3 indices.memory.index\_buffer\_size: 20% indices.fielddata.cache.size: 10% gateway.expected\_nodes: 1 cluster.routing.allocation.allow\_rebalance: indices\_primaries\_active node.attr.sitename: xxx discovery.zen.ping\_timeout: 10s gateway.recover\_after\_nodes: 1 gateway.recover\_after\_time: 5m discovery.zen.minimum\_master\_nodes: 2 ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 1:15pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/10 "2018-05-18T13:15:59Z")
I've also set the minimum \_master node to 2 and restarted one node. They can not find each other... It works only if I restart both nodes... ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [May 18, 2018, 2:23pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/11 "2018-05-18T14:23:02Z")
after a yaml config change (on all nodes) a restart must be actioned, on all nodes. ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 18, 2018, 2:52pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/12 "2018-05-18T14:52:30Z")
Thank you for your feedback. I'm curious to know if I can have a two nodes cluster and which can work even one node crash. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 18, 2018, 3:40pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/13 "2018-05-18T15:40:05Z")
If you are looking for high availability you need a minimum of 3 master-eligible nodes so the remaining two nodes can form a majority and elect a new master. ---
**Author:** ![Abhilash\_Bolla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abhilash_bolla/32/26840_2.png) [@Abhilash\_Bolla](https://discuss.elastic.co/u/Abhilash_Bolla)\ **Post date:** [May 21, 2018, 11:35am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/14 "2018-05-21T11:35:10Z")
It'll work only if the slave node is down. AFAIK, Re-election will take place only if both the nodes are master-eligible. ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 28, 2018, 9:21am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/15 "2018-05-28T09:21:35Z")
These are the configuration of three nodes: node1: ![node1](https://us1.discourse-cdn.com/elastic/original/3X/a/a/aa529fee3a65a89f08d4bb5dcc1d5274fe70c415.png) node2: ![node2](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1d9280fabc5a8b8c928d1746e626929ca19c905d.png) node3: ![node3](https://us1.discourse-cdn.com/elastic/original/3X/4/9/49587f6237716feae26b118334d4467ee31d2956.png) After restarting the master node (node 1), the two rmaining nodes doesn't respond to any request. In the logs of Node3 there are logs which saying node3 still trying to connect to gone master (node 1), instead of forming a new cluster with node 2, and the same for node 3. Thank you for your feedbacks. Vahid ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 28, 2018, 12:59pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/16 "2018-05-28T12:59:03Z")
Both are master-eligible. Anyway it seems that high availibility with two nodes with the versions higher that 1.x is not supported any more and at least three nodes must be in the cluster. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 28, 2018, 3:11pm UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/17 "2018-05-28T15:11:04Z")
It has never been possible to have a fully highly available cluster with only two nodes. ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 29, 2018, 6:29am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/18 "2018-05-29T06:29:54Z")
It was possible, we were using it for our test environments and after crashing one node, still cluster was serving properly with only one node. However with suffering from split-brain sometimes, even with three nodes... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2018, 6:56am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/19 "2018-05-29T06:56:56Z")
If you were able to serve writes when one node was down in a two-node cluster, it means that you have not set `minimum_master_nodes` correctly. This can lead to split-brain scenarios and data loss. You should always set `minimum_master_nodes` according to [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-node.html#split-brain). ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [May 29, 2018, 7:10am UTC](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455/20 "2018-05-29T07:10:12Z")
As I wrote in the initial comment, we were using ES 1.7 and for this version always there was a possibility of a split brain, even with configuring the minimum\_mast\_nodes to a high value (nodes/2 +1) and more than two nodes. Have a look to the old version ([https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-node.html#split-brain](https://www.elastic.co/guide/en/elasticsearch/reference/1.7/modules-node.html#split-brain)). Howver now we have a bigger problem, with three nodes and mentioned configuration we have no high availibility! [Next page](https://discuss.elastic.co/t/could-not-get-cluster-status-after-master-node-goes-down/132455.md?page=2)