# Failed to join a cluster because of time-out **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2020, 2:29am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360 "2020-06-24T02:29:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dasolyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dasolyn/32/72747_2.png) [@dasolyn](https://discuss.elastic.co/u/dasolyn)\ **Post date:** [June 24, 2020, 2:29am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/1 "2020-06-24T02:29:33Z")
I'm trying to build ElasticSearch cluster but it cause an error. Log for master node (All IPs and comments were omitted due to privacy) ```auto [2020-06-23T16:33:47,361][WARN][o.e.c.c.Coordinator] [kn-log-01] failed to validate incoming join request from node [{kn-log-02}{tuCA1_YARK-HkHyzbpG4Nw}{0yZHEJGAQpKgWw336U2vDQ}{127.0.0.2}{127.0.0.2:9300}{dilrt}{ml.machine_memory=134888939520, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] org.elasticsearch.transport.ReceiveTimeoutTransportException: [kn-log-02][127.0.0.2:9300][internal:cluster/coordination/join/validate] request_id [88] timed out after [59835ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1041) [elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.7.0.jar:7.7.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] ``` Log for data node to join ```auto org.elasticsearch.transport.RemoteTransportException: [kn-log-01][127.0.0.1:9300][internal:cluster/coordination/join] Caused by: java.lang.IllegalStateException: failure when sending a validation request to node at org.elasticsearch.cluster.coordination.Coordinator$2.onFailure(Coordinator.java:514) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.transport.TransportService$8.run(TransportService.java:1001) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.7.0.jar:7.7.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] Caused by: org.elasticsearch.transport.NodeDisconnectedException: [kn-log-02][127.0.0.2:9300][internal:cluster/coordination/join/validate] disconnected [2020-06-23T16:41:47,433][WARN][o.e.c.c.ClusterFormationFailureHelper] [kn-log-02] master not discovered yet: have discovered [{kn-log-02}{tuCA1_YARK-HkHyzbpG4Nw}{0yZHEJGAQpKgWw336U2vDQ}{127.0.0.2}{127.0.0.2:9300}{dilrt}{ml.machine_memory=134888939520, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.3:9300, 127.0.0.4:9300] from hosts providers and [] from last-known cluster state; node term 1, last-accepted version 0 in term 0 [2020-06-23T16:41:57,434][WARN][o.e.c.c.ClusterFormationFailureHelper] [kn-log-02] master not discovered yet: have discovered [{kn-log-02}{tuCA1_YARK-HkHyzbpG4Nw}{0yZHEJGAQpKgWw336U2vDQ}{127.0.0.2}{127.0.0.2:9300}{dilrt}{ml.machine_memory=134888939520, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.3:9300, 127.0.0.4:9300] from hosts providers and [] from last-known cluster state; node term 1, last-accepted version 0 in term 0 ``` The node trying to request joining every minutes but caues time-out error. It doesn't work now but yesterday did without changing any settings about ElasticSearch (maybe). elasticsearch.yml for master node ```auto cluster.name: mycluster node.name: kn-log-01 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 discovery.seed_hosts: ["127.0.0.1", "127.0.0.2", "127.0.0.3", "127.0.0.4"] cluster.initial_master_nodes: ["kn-log-01"] node.master: true node.data: true ``` elasticsearch.yml for data node ```auto cluster.name: mycluster node.name: kn-log-02 path.data: /data/elasticsearch path.logs: /var/log/elasticsearch network.host: 0.0.0.0 discovery.seed_hosts: ["127.0.0.1", "127.0.0.2", "127.0.0.3", "127.0.0.4"] cluster.initial_master_nodes: ["kn-log-01"] node.master: false node.data: true ``` ```auto $ curl -XGET 127.0.0.1:9200 { "name" : "kn-log-01", "cluster_name" : "mycluster", "cluster_uuid" : "jN-0FJwDRZqlAtQ6LpXwug", "version" : { "number" : "7.7.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf", "build_date" : "2020-05-12T02:01:37.602180Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } $ curl -XGET 127.0.0.2:9200 { "name" : "kn-log-02", "cluster_name" : "mycluster", "cluster_uuid" : "_na_", "version" : { "number" : "7.7.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf", "build_date" : "2020-05-12T02:01:37.602180Z", "build_snapshot" : false, "lucene_version" : "8.5.1", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" } $ curl -XGET 127.0.0.1:9200/_cat/nodes?v ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 127.0.0.1 15 2 0 0.01 0.03 0.05 dilmrt * kn-log-01 ``` What I did already: - Checking firewalld settings about 9200, 9300 port again. - Rebooting all machines. - Wipe ElasticSearch data folders and restart services. ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [June 24, 2020, 3:17am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/2 "2020-06-24T03:17:17Z")
> [@dasolyn](#): > > ` ` Could you telnet other x.x.x.x 9300 on each host? ---
**Author:** ![dasolyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dasolyn/32/72747_2.png) [@dasolyn](https://discuss.elastic.co/u/dasolyn)\ **Post date:** [June 24, 2020, 4:06am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/3 "2020-06-24T04:06:59Z")
Telnet 9300 port working well including to master node and to data node when ElasticSearch service turned on. ---
**Author:** ![dasolyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dasolyn/32/72747_2.png) [@dasolyn](https://discuss.elastic.co/u/dasolyn)\ **Post date:** [June 25, 2020, 5:22am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/4 "2020-06-25T05:22:19Z")
When I changed kn-log-02 to master eligible node, I got a following log. ```auto [2020-06-25T14:10:13,181][WARN][o.e.t.TransportService] [kn-log-02] Received response for a request that has timed out, sent [60036ms] ago, timed out [0ms] ago, action [internal:cluster/coordination/join], node [{kn-log-01}{2vdl6zdaTDWFjeGc9D6boQ}{8TNI9H74SFK1lKxH0VoTxA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134888939520, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}], id [129] [2020-06-25T14:10:22,139][INFO][o.e.c.c.JoinHelper] [kn-log-02] last failed join attempt was 8.9s ago, failed to join {kn-log-01}{2vdl6zdaTDWFjeGc9D6boQ}{8TNI9H74SFK1lKxH0VoTxA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134888939520, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} with JoinRequest{sourceNode={kn-log-02}{o7ei8kSHTLGaLlTXxGzenw}{KkL6tshFSM6lmS5by9CIbw}{127.0.0.2}{127.0.0.2:9300}{dilmrt}{ml.machine_memory=134888873984, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=1, optionalJoin=Optional[Join{term=1, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={kn-log-02}{o7ei8kSHTLGaLlTXxGzenw}{KkL6tshFSM6lmS5by9CIbw}{127.0.0.2}{127.0.0.2:9300}{dilmrt}{ml.machine_memory=134888873984, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, targetNode={kn-log-01}{2vdl6zdaTDWFjeGc9D6boQ}{8TNI9H74SFK1lKxH0VoTxA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134888939520, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}}]} org.elasticsearch.transport.ReceiveTimeoutTransportException: [kn-log-01][127.0.0.1:9300][internal:cluster/coordination/join] request_id [129] timed out after [60036ms] at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1041) ~[elasticsearch-7.7.0.jar:7.7.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) ~[elasticsearch-7.7.0.jar:7.7.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?] at java.lang.Thread.run(Thread.java:832) [?:?] [2020-06-25T14:10:22,141][WARN][o.e.c.c.ClusterFormationFailureHelper] [kn-log-02] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [kn-log-01] to bootstrap a cluster: have discovered [{kn-log-02}{o7ei8kSHTLGaLlTXxGzenw}{KkL6tshFSM6lmS5by9CIbw}{127.0.0.2}{127.0.0.2:9300}{dilmrt}{ml.machine_memory=134888873984, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {kn-log-01}{2vdl6zdaTDWFjeGc9D6boQ}{8TNI9H74SFK1lKxH0VoTxA}{127.0.0.1}{127.0.0.1:9300}{dilmrt}{ml.machine_memory=134888939520, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}]; discovery will continue using [127.0.0.1:9300, 127.0.0.3:9300, 127.0.0.4:9300] from hosts providers and [{kn-log-02}{o7ei8kSHTLGaLlTXxGzenw}{KkL6tshFSM6lmS5by9CIbw}{127.0.0.2}{127.0.0.2:9300}{dilmrt}{ml.machine_memory=134888873984, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 1, last-accepted version 0 in term 0 ``` I feel it is very strange. The log says `this node must discover kn-log-01 to bootstrap: but have discoverd [kn-log-02, kn-log-01]` ---
**Author:** ![dasolyn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dasolyn/32/72747_2.png) [@dasolyn](https://discuss.elastic.co/u/dasolyn)\ **Post date:** [June 26, 2020, 4:55am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/5 "2020-06-26T04:55:50Z")
I re-installed ELK stacks now server #1 and #2 can build a cluster but #3 and #4 can't. I got a new log message when trying to connect 3rd, 4th server. ```auto [2020-06-26T12:48:28,802][WARN][o.e.t.TcpTransport] [kn-log-01] invalid internal transport message format, got (ff,f4,ff,fd), [Netty4TcpChannel{localAddress=/127.0.0.1:9300, remoteAddress=/127.0.0.4:51254}], closing connection ``` This log doesn't appear before reinstall. I'm not sure this log related the issue but I'm trying to resolve from here. ---
**Author:** ![wangqinghuan](https://avatars.discourse-cdn.com/v4/letter/w/d26b3c/32.png) [@wangqinghuan](https://discuss.elastic.co/u/wangqinghuan)\ **Post date:** [June 26, 2020, 2:56pm UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/6 "2020-06-26T14:56:44Z")
Sorry, not know what's going on. There is an [issue](https://github.com/elastic/elasticsearch/issues/12355) about this log message. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 21, 2020, 8:51am UTC](https://discuss.elastic.co/t/failed-to-join-a-cluster-because-of-time-out/238360/8 "2020-08-21T08:51:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.