# How many master node should be in configuration **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [April 25, 2022, 8:15pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198 "2022-04-25T20:15:42Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 8:15pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/1 "2022-04-25T20:15:42Z")
Hi I've developed such configuration host1 2 x master node 3x data hot 9x data warm host2 2 x master node 3x data hot 9x data warm in the docker logs I don't see any suspicious but I can't set up a kibana\_system nor I didn't get info from ```auto curl -k http://10.244.12.241:9202/_cat/master?v -u elastic Enter host password for user 'elastic': {"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}[elasticsearch@srv24246anm-kvm kickstart_elk_cluster]$ ``` What's wrong? ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 9:35pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/2 "2022-04-25T21:35:12Z")
I saw such things ```auto {"@timestamp":"2022-04-25T21:34:07.340Z", "log.level": "WARN", "message":"master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es_master_1_1, es_master_2_1, es_master_1_2, es_master_2_2] to bootstrap a cluster: have discovered [{es_master_2_1}{8nM0u-YoR9ieVudd9WT9mA}{idycjo8tTrG7pyRtQET-TQ}{10.0.0.95}{10.0.0.95:9300}{m}]; discovery will continue using [10.0.9.45:9300, 10.0.9.95:9300, 10.0.9.105:9300, 10.0.9.78:9300, 10.0.9.68:9300, 10.0.9.56:9300, 10.0.9.64:9300, 10.0.9.60:9300, 10.0.9.48:9300, 10.0.9.52:9300, 10.0.9.70:9300, 10.0.9.74:9300, 10.0.9.76:9300, 10.0.9.58:9300, 10.0.9.50:9300, 10.0.9.66:9300, 10.0.9.62:9300, 10.0.9.54:9300, 10.0.9.113:9300, 10.0.9.91:9300, 10.0.9.83:9300, 10.0.9.107:9300, 10.0.9.101:9300, 10.0.9.87:9300, 10.0.9.85:9300, 10.0.9.109:9300, 10.0.9.80:9300, 10.0.9.99:9300, 10.0.9.97:9300, 10.0.9.93:9300, 10.0.9.103:9300, 10.0.9.111:9300, 10.0.9.89:9300] from hosts providers and [{es_master_2_1}{8nM0u-YoR9ieVudd9WT9mA}{idycjo8tTrG7pyRtQET-TQ}{10.0.0.95}{10.0.0.95:9300}{m}] from last-known cluster state; node term 0, last-accepted version 0 in term 0", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_2_1][generic][T#11]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es_master_2_1","elasticsearch.cluster.name":"elk_cluster"} ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 9:35pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/3 "2022-04-25T21:35:54Z")
but my config looks good: ```auto - cluster.name=elk_cluster - discovery.seed_hosts=es_master_2_1,es_master_1_2,es_master_2_2,es_data_ssd_3_1_ingest,es_data_ssd_1_1,es_data_ssd_2_1,es_data_ssd_3_1,es_data_ssd_4_1,es_data_ssd_5_1,es_data_hdd_1_1,es_data_hdd_2_1,es_data_hdd_3_1,es_data_hdd_4_1,es_data_hdd_5_1,es_data_hdd_6_1,es_data_hdd_7_1,es_data_hdd_8_1,es_data_hdd_9_1,es_data_ssd_3_2_ingest,es_data_ssd_1_2,es_data_ssd_2_2,es_data_ssd_3_2,es_data_ssd_4_2,es_data_ssd_5_2,es_data_hdd_1_2,es_data_hdd_2_2,es_data_hdd_3_2,es_data_hdd_4_2,es_data_hdd_5_2,es_data_hdd_6_2,es_data_hdd_7_2,es_data_hdd_8_2,es_data_hdd_9_2 - cluster.initial_master_nodes=es_master_1_1, es_master_2_1, es_master_1_2, es_master_2_2 ``` ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 25, 2022, 9:40pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/4 "2022-04-25T21:40:52Z")
to be safe you need three because even number node can create split brain situation. ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 10:31pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/5 "2022-04-25T22:31:49Z")
so I've extended master nodes from 2 masters per host to 3 per hosts but still got ```auto {"@timestamp":"2022-04-25T22:30:59.772Z", "log.level": "WARN", "message":"master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [es_master_1_1, es_master_2_1, es_master_1_2, es_master_2_2, es_master_3_1, es_master_3_2] to bootstrap a cluster: have discovered [{es_master_1_1}{q23IasYaQse8GgbYhQ8QVA}{Ax60i-OBQEqeqXriOeGypQ}{10.0.0.206}{10.0.0.206:9300}{m}]; discovery will continue using [10.0.9.170:9300, 10.0.9.223:9300, 10.0.9.219:9300, 10.0.9.188:9300, 10.0.9.184:9300, 10.0.9.186:9300, 10.0.9.178:9300, 10.0.9.164:9300, 10.0.9.161:9300, 10.0.9.176:9300, 10.0.9.194:9300, 10.0.9.172:9300, 10.0.9.190:9300, 10.0.9.174:9300, 10.0.9.180:9300, 10.0.9.168:9300, 10.0.9.196:9300, 10.0.9.166:9300, 10.0.9.198:9300, 10.0.9.205:9300, 10.0.9.213:9300, 10.0.9.221:9300, 10.0.9.209:9300, 10.0.9.211:9300, 10.0.9.207:9300, 10.0.9.227:9300, 10.0.9.233:9300, 10.0.9.201:9300, 10.0.9.203:9300, 10.0.9.231:9300, 10.0.9.217:9300, 10.0.9.225:9300, 10.0.9.215:9300, 10.0.9.182:9300, 10.0.9.229:9300] from hosts providers and [{es_master_1_1}{q23IasYaQse8GgbYhQ8QVA}{Ax60i-OBQEqeqXriOeGypQ}{10.0.0.206}{10.0.0.206:9300}{m}] from last-known cluster state; node term 0, last-accepted version 0 in term 0", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_1_1][generic][T#8]","log.logger":"org.elasticsearch.cluster.coordination.ClusterFormationFailureHelper","elasticsearch.node.name":"es_master_1_1","elasticsearch.cluster.name":"elk_cluster"} ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 10:47pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/6 "2022-04-25T22:47:16Z")
and I'm pretty sure that we don't have any firewall block from iptables it seems that we have already opened ports ```auto Chain DOCKER-INGRESS (1 references) target prot opt source destination ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9331 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9331 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9231 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9231 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9334 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9334 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9234 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9234 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9339 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9339 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9239 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9239 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9330 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9330 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9230 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9230 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9336 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9336 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9236 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9236 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9320 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9320 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9220 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:9220 ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:9326 ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 10:49pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/7 "2022-04-25T22:49:32Z")
uff after 1 minute (it took discovery process) were connected 🙂 ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [April 25, 2022, 11:52pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/8 "2022-04-25T23:52:37Z")
now just make sure all is there curl -u user:password -XGET master\_name:9200/\_cat/nodes?v ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 11:57pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/9 "2022-04-25T23:57:08Z")
> [@elasticforme](#): > > curl -u user:password -XGET master\_name:9200/\_cat/nodes?v curl -u elastic:password -XGET [http://10.250.131.225:9202/\_cat/nodes?v](http://10.250.131.225:9202/_cat/nodes?v) {"error":{"root\_cause":[{"type":"master\_not\_discovered\_exception","reason":null}],"type":"master\_not\_discovered\_exception","reason":null},"status":503}[elasticsearch@srv24246anm-kvm kickstart\_elk\_cluster]$ ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 11:58pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/10 "2022-04-25T23:58:40Z")
in the meantime I find that: ```auto {"@timestamp":"2022-04-25T23:53:01.623Z", "log.level": "WARN", "message":"[connectToRemoteMasterNode[10.0.9.187:9300]] completed handshake with [{es_master_3_2}{utHeMP5oQGeb4VWyZHBh4g}{1LZOSe_PSI-3PGbGpWlgiA}{10.0.0.197}{10.0.0.197:9300}{m}{xpack.installed=true}] but followup connection failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#13]","log.logger":"org.elasticsearch.discovery.HandshakingTransportAddressConnector","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[es_master_3_2][10.0.0.197:9300] connect_timeout[30s]","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [es_master_3_2][10.0.0.197:9300] connect_timeout[30s]\n\tat org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1113)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"} {"@timestamp":"2022-04-25T23:53:19.762Z", "log.level": "WARN", "message":"[connectToRemoteMasterNode[10.0.9.136:9300]] completed handshake with [{es_master_1_1}{q23IasYaQse8GgbYhQ8QVA}{6vIGdALqQWmD4g6g8OUAkQ}{10.0.0.147}{10.0.0.147:9300}{m}{xpack.installed=true}] but followup connection failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#12]","log.logger":"org.elasticsearch.discovery.HandshakingTransportAddressConnector","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[es_master_1_1][10.0.0.147:9300] connect_timeout[30s]","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [es_master_1_1][10.0.0.147:9300] connect_timeout[30s]\n\tat org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1113)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"} {"@timestamp":"2022-04-25T23:53:29.836Z", "log.level": "WARN", "message":"[connectToRemoteMasterNode[10.0.9.183:9300]] completed handshake with [{es_master_2_2}{29iHv6-bQPKrbuh84GcElA}{_Pj7x2MSQCWw8tL9eYyFVw}{10.0.0.193}{10.0.0.193:9300}{m}{xpack.installed=true}] but followup connection failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#15]","log.logger":"org.elasticsearch.discovery.HandshakingTransportAddressConnector","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[es_master_2_2][10.0.0.193:9300] connect_timeout[30s]","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [es_master_2_2][10.0.0.193:9300] connect_timeout[30s]\n\tat org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1113)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"} ``` but now discover result looks good ```auto {"@timestamp":"2022-04-25T23:58:09.185Z", "log.level": "WARN", "message":"address [10.0.9.154:9300], node [null], requesting [false] discovery result: [es_data_hdd_3_1][10.0.0.165:9300] successfully discovered master-ineligible node {es_data_hdd_3_1}{tCHKsyhdTgaHa5b_JtVuAw}{tX_Un7PkR9S9QJfq7xRlVA}{10.0.0.165}{10.0.0.165:9300}{w} at [10.0.9.154:9300]; to suppress this message, remove address [10.0.9.154:9300] from your discovery configuration or ensure that traffic to this address is routed only to master-eligible nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#31]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster"} {"@timestamp":"2022-04-25T23:58:09.187Z", "log.level": "WARN", "message":"address [10.0.9.158:9300], node [null], requesting [false] discovery result: [es_data_ssd_1_1][10.0.0.169:9300] successfully discovered master-ineligible node {es_data_ssd_1_1}{U-tbIYFYSX63MZFdleIi4A}{xb9PROnHSau9PaVsGeWaWg}{10.0.0.169}{10.0.0.169:9300}{hs} at [10.0.9.158:9300]; to suppress this message, remove address [10.0.9.158:9300] from your discovery configuration or ensure that traffic to this address is routed only to master-eligible nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#8]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster"} {"@timestamp":"2022-04-25T23:58:09.188Z", "log.level": "WARN", "message":"address [10.0.9.144:9300], node [null], requesting [false] discovery result: [es_data_ssd_3_1][10.0.0.155:9300] successfully discovered master-ineligible node {es_data_ssd_3_1}{eqPk7f-vTGGHkxrAxJmAFQ}{-1Ix2kadTGa0bQBmCLSDSw}{10.0.0.155}{10.0.0.155:9300}{hs} at [10.0.9.144:9300]; to suppress this message, remove address [10.0.9.144:9300] from your discovery configuration or ensure that traffic to this address is routed only to master-eligible nodes", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_3_1][generic][T#23]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.node.name":"es_master_3_1","elasticsearch.cluster.name":"elk_cluster"} ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 25, 2022, 11:59pm UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/11 "2022-04-25T23:59:27Z")
so something is still left .... ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 26, 2022, 12:02am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/12 "2022-04-26T00:02:36Z")
suggestions for advanced troubleshooting appreciated ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 26, 2022, 12:04am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/13 "2022-04-26T00:04:01Z")
curl -u elastic:o8toW7nf64DAy3 -XGET [http://10.250.131.225:9202](http://10.250.131.225:9202) { "name" : "es\_master\_1\_1", "cluster\_name" : "elk\_cluster", "cluster\_uuid" : "_na_", "version" : { "number" : "8.1.0", "build\_flavor" : "default", "build\_type" : "docker", "build\_hash" : "3700f7679f7d95e36da0b43762189bab189bc53a", "build\_date" : "2022-03-03T14:20:00.690422633Z", "build\_snapshot" : false, "lucene\_version" : "9.0.0", "minimum\_wire\_compatibility\_version" : "7.17.0", "minimum\_index\_compatibility\_version" : "7.0.0" }, "tagline" : "You Know, for Search" } ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 26, 2022, 12:43am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/14 "2022-04-26T00:43:54Z")
what can I more explain , 10.0.0.5:9300 this IP comes from docker swarm `ingress` network, so in this point we have a problem I think so but "[10.0.9.2:9300]] completed handshake with" comes from `mynet` network and it works at least docker swarm network is issuer ```auto {"@timestamp":"2022-04-26T00:40:01.104Z", "log.level": "WARN", "message":"[connectToRemoteMasterNode[10.0.9.2:9300]] completed handshake with [{es_master_1_1}{q23IasYaQse8GgbYhQ8QVA}{5tpB5uS9R12MOqrBqlXIgw}{10.0.0.5}{10.0.0.5:9300}{m}{xpack.installed=true}] but followup connection failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[es_master_2_1][generic][T#23]","log.logger":"org.elasticsearch.discovery.HandshakingTransportAddressConnector","elasticsearch.node.name":"es_master_2_1","elasticsearch.cluster.name":"elk_cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[es_master_1_1][10.0.0.5:9300] connect_timeout[30s]","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [es_master_1_1][10.0.0.5:9300] connect_timeout[30s]\n\tat org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1113)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:717)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"} {"@timestamp":"2022-04-26T00:40:07.127Z", "log.level": "WARN", "message":"[connectToRemoteMasterNode[10.0.9.48:9300]] ``` ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 26, 2022, 1:21am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/15 "2022-04-26T01:21:14Z")
```auto elasticsearch@bde3eeaf65bc:~$ curl -v http://10.0.9.60:9300 * Trying 10.0.9.60:9300... * TCP_NODELAY set * Connected to 10.0.9.60 (10.0.9.60) port 9300 (#0) > GET / HTTP/1.1 > Host: 10.0.9.60:9300 > User-Agent: curl/7.68.0 > Accept: */* > * Empty reply from server * Connection #0 to host 10.0.9.60 left intact curl: (52) Empty reply from server elasticsearch@bde3eeaf65bc:~$ curl -v http://10.0.0.61:9300 * Trying 10.0.0.61:9300... * TCP_NODELAY set ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [April 26, 2022, 3:13am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/16 "2022-04-26T03:13:34Z")
> [@INS](#): > > host1 > 2 x master node > 3x data hot > 9x data warm > > host2 > 2 x master node > 3x data hot > 9x data warm > [@INS](#): > > so I've extended master nodes from 2 masters per host to 3 per hosts If you are looking for high availability you will need three hosts. Having 3 master nodes per host instead of 2 will not help. ---
**Author:** ![INS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ins/32/92827_2.png) [@INS](https://discuss.elastic.co/u/INS)\ **Post date:** [April 26, 2022, 8:44am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/17 "2022-04-26T08:44:48Z")
I found the soultion if You have a problem with ingress this is short workaround: ```auto 1.docker swarm init --advertise-addr 10.244.12.241 2. `docker network rm ingress` 3. docker network create \ --driver overlay \ --ingress \ --subnet=10.255.0.0/16 \ --gateway=10.255.0.1 \ my-ingress 4. systemctl restart docker ``` it was tested on Docker version 20.10.14, build a224086 and Elasticsearch in 8.1.3 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 24, 2022, 8:45am UTC](https://discuss.elastic.co/t/how-many-master-node-should-be-in-configuration/303198/18 "2022-05-24T08:45:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.