# ES 2.4.6 Exception caught on transport layer : connection timeout? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 1, 2019, 11:26am UTC](https://discuss.elastic.co/t/es-2-4-6-exception-caught-on-transport-layer-connection-timeout/188298 "2019-07-01T11:26:22Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![wuxiansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wuxiansen/32/49237_2.png) [@wuxiansen](https://discuss.elastic.co/u/wuxiansen)\ **Post date:** [July 1, 2019, 11:26am UTC](https://discuss.elastic.co/t/es-2-4-6-exception-caught-on-transport-layer-connection-timeout/188298/1 "2019-07-01T11:26:22Z")
### Elasticsearch version\*\* : 2.4.6 ### JVM version\*\* : ```auto java version "1.8.0_144" ### OS version** : Centos 64 ``` ### elastic yml : ```auto ···· gateway.recover_after_nodes: 2 gateway.expected_nodes: 2 gateway.recover_after_time: 5m discovery.zen.fd.ping_interval: 1s discovery.zen.fd.ping_timeout: 30s discovery.zen.fd.ping_retries: 3 discovery.zen.ping.timeout: 30s discovery.zen.ping.multicast.enabled: false discovery.zen.minimum_master_nodes: 2 ``` ### cat/node ```auto node-3.es-pro.com 192.168.254.239 67 99 1.12 d - node-03-239 node-1.es-pro.com 192.168.254.237 60 100 2.97 d - node-01-237 node-0.es-pro.com 192.168.254.236 5 60 0.27 - - client-node-236 node-7.es-pro.com 192.168.250.215 71 95 0.24 d - node-07-215 node-5.es-pro.com 192.168.254.214 62 100 3.13 d m node-05-214 node-2.es-pro.com 192.168.254.238 50 97 1.65 d - node-02-238 node-4.es-pro.com 192.168.254.215 43 100 3.00 d m node-04-215 node-8.es-pro.com 192.168.250.216 64 100 2.56 d - node-08-216 node-6.es-pro.com 192.168.254.213 69 99 1.89 d * node-06-213 ``` ### ( 3 num )Master Log: ```auto [2019-06-28 20:36:47,136][WARN][cluster.action.shard] [node-06-213] [ffffffffn_2016][0] received shard failed for target shard [[ffffffffn_2016][0], node[6wfUIeWlTfGmxADxqdBsRw], relocating [yZkxF7GiTVW9CQm4NQBGjA], [P], v[237], s[INITIALIZING], a[id=bHW3WXeZQYmoNsO391yzWQ, rId=g57zPop3TZu5iyhqcwNcVA], expected_shard_size[58285273234]], indexUUID [xJ6n1X8bREWws63uZHBb4g], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]] NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] [2019-06-28 20:36:57,988][WARN][transport.netty] [node-06-213] exception caught on transport layer [[id: 0x3a951ccb, /192.168.250.216:57213 => /192.168.254.213:6300]], closing connection java.io.IOException: connection timeout at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:192) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [2019-06-28 20:37:32,482][INFO][cluster.service] [node-06-213] added {{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false},}, reason: zen-disco-join(join from node[{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}]) [2019-06-28 20:39:47,942][INFO][cluster.routing.allocation] [node-06-213] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ffffffffn_2016][0]] ...]). [2019-06-28 20:42:24,074][WARN][cluster.action.shard] [node-06-213] [ffffffffn_2016][1] received shard failed for target shard [[ffffffffn_2016][1], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[131], s[STARTED], a[id=Vz_0RHCyS3eC7AV70Xcxwg]], indexUUID [grryTVt0Rr2NY3Jrsq2MHw], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]] NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] [2019-06-28 20:42:24,074][WARN][cluster.action.shard] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]] NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] [2019-06-28 20:42:24,074][WARN][cluster.action.shard] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]] NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] [2019-06-28 20:42:27,583][WARN][action.bulk] [node-06-213] [invoice_detail_2015][0] failed to perform indices:data/write/bulk[s][r] on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false} NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] [2019-06-28 20:42:27,583][WARN][cluster.action.shard] [node-06-213] [invoice_detail_2015][0] received shard failed for target shard [[invoice_detail_2015][0], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[221], s[STARTED], a[id=wxsACNOMR_SGO8i7WTcDMQ]], indexUUID [HRzC8xbsQxuON4lmvH1c3A], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]] NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] ········ ``` **( 8 num )Node Log**: ### Problem: I don't know why it's been added and removed then added and deleted again..again .········ ?\*\* > ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 5, 2019, 9:52am UTC](https://discuss.elastic.co/t/es-2-4-6-exception-caught-on-transport-layer-connection-timeout/188298/2 "2019-07-05T09:52:48Z")
> [@wuxiansen](#): > > `java.io.IOException: connection timeout` This indicates a problem with your network. It's hard to give more specific advice about a version as old as 2.4.6, except to recommend that you upgrade to a more recent version. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2019, 9:52am UTC](https://discuss.elastic.co/t/es-2-4-6-exception-caught-on-transport-layer-connection-timeout/188298/3 "2019-08-02T09:52:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.