# Elasticsearch Node failed to send join request to master **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2019, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-node-failed-to-send-join-request-to-master/193009 "2019-07-31T01:10:06Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![xiang-liu-exa](https://avatars.discourse-cdn.com/v4/letter/x/71e660/32.png) [@xiang-liu-exa](https://discuss.elastic.co/u/xiang-liu-exa)\ **Post date:** [July 31, 2019, 1:10am UTC](https://discuss.elastic.co/t/elasticsearch-node-failed-to-send-join-request-to-master/193009/1 "2019-07-31T01:10:06Z")
Elasticsearch version 6.2.4 Cluster: 46 Nodes, Master: host1-0 Cluster is running for 4 days without any problem. About 10 hour ago, one data Node: host9-0 fail to ping master After that, host9-0 keeps send join request to master but get "a channel closed while connecting" error. Here is the log (if needed, I can provide more logs) > Blockquote > [2019-07-30T14:45:38,919][INFO][o.e.d.z.ZenDiscovery] [host9-0] master\_left [{host1-0}{XnTUIM0gRGuYPDo07kb7rw}{WhOIwzlvTfuU3O3mAxq88A}{elasticsearch-host1-a}{10.50.55.143:9300}{rack\_id=host1}], reason [failed to ping, tried [3] times, each with maximum [1m] timeout] > [2019-07-30T14:45:38,985][WARN][o.e.d.z.ZenDiscovery] [host9-0] master left (reason = failed to ping, tried [3] times, each with maximum [1m] timeout), current nodes: nodes: > ... > [2019-07-30T14:45:39,166][WARN][o.e.c.NodeConnectionsService] [host9-0] failed to connect to node {host4-0}{zEqozy7ARYaH9gk5vyCGDg}{udD\_q1b5Tz-EEg7Izx4y7g}{elasticsearch-host4-a}{10.50.52.174:9300}{rack\_id=host4, box\_type=hot} (tried [1] times) > org.elasticsearch.transport.ConnectTransportException: [host4-0][10.50.52.174:9300] a channel closed while connecting > at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:652) ~[elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4] > at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_91] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_91] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_91] > [2019-07-30T14:45:39,238][WARN][c.f.s.s.t.SearchGuardSSLNettyTransport] [host9-0] exception caught on transport layer [NettyTcpChannel{localAddress=/10.50.53.114:42702, remoteAddress=elasticsearch-host1-d/10.50.55.180:9300}], closing connection > java.lang.IllegalArgumentException: null > at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0\_91] > at io.netty.buffer.PooledHeapByteBuf.setBytes(PooledHeapByteBuf.java:261) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final] > at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343) ~[netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) [netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] > at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_91] > [2019-07-30T14:45:39,358][WARN][i.n.c.AbstractChannelHandlerContext] Failed to mark a promise as failure because it has succeeded already: DefaultChannelPromise@61947277(success) > java.lang.IndexOutOfBoundsException: null > at java.nio.ByteBuffer.wrap(ByteBuffer.java:375) ~[?:1.8.0\_91] > at io.netty.buffer.PooledHeapByteBuf.nioBuffer(PooledHeapByteBuf.java:300) ~[netty-buffer-4.1.16.Final.jar:4.1.16.Final] > ...... > Jul 31 00:27:50 psr-169-56 docker[3817]: [2019-07-31T00:27:50,978][WARN][c.f.s.s.t.SearchGuardSSLNettyTransport] [host9-0] exception caught on transport layer [NettyTcpChannel{localAddress=/10.50.53.114:9300, remoteAddress=/10.50.55.143:50974}], closing connection > Jul 31 00:27:50 psr-169-56 docker[3817]: java.lang.IllegalArgumentException: null > Jul 31 00:27:50 psr-169-56 docker[3817]: at java.nio.Buffer.position(Buffer.java:244) ~[?:1.8.0\_91] > ...... > Jul 31 00:27:51 psr-169-56 docker[3817]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] > Jul 31 00:27:51 psr-169-56 docker[3817]: at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_91] > Jul 31 00:27:51 psr-169-56 docker[3817]: [2019-07-31T00:27:51,207][INFO][o.e.d.z.ZenDiscovery] [host9-0] failed to send join request to master [{host1-0}{XnTUIM0gRGuYPDo07kb7rw}{WhOIwzlvTfuU3O3mAxq88A}{elasticsearch-host1-a}{10.50.55.143:9300}{rack\_id=host1}], reason [RemoteTransportException[[host1-0][10.50.55.143:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[host9-0][10.50.53.114:9300] a channel closed while connecting]; ] ---
**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Post date:** [July 31, 2019, 5:16am UTC](https://discuss.elastic.co/t/elasticsearch-node-failed-to-send-join-request-to-master/193009/2 "2019-07-31T05:16:37Z")
If you do a telnet host1-0 9200 from host9-0 does that work? ---
**Author:** ![xiang-liu-exa](https://avatars.discourse-cdn.com/v4/letter/x/71e660/32.png) [@xiang-liu-exa](https://discuss.elastic.co/u/xiang-liu-exa)\ **Post date:** [July 31, 2019, 3:34pm UTC](https://discuss.elastic.co/t/elasticsearch-node-failed-to-send-join-request-to-master/193009/3 "2019-07-31T15:34:28Z")
Yes, both 9200 and 9300 work. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2019, 3:34pm UTC](https://discuss.elastic.co/t/elasticsearch-node-failed-to-send-join-request-to-master/193009/4 "2019-08-28T15:34:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.