ES 2.4.6 Exception caught on transport layer : connection timeout?

Elastic Stack Elasticsearch
1

Elasticsearch version** : 2.4.6

JVM version** :

java version "1.8.0_144"

### OS version**  : Centos 64

elastic yml :

····

gateway.recover_after_nodes: 2 
gateway.expected_nodes: 2
gateway.recover_after_time: 5m 
discovery.zen.fd.ping_interval: 1s 
discovery.zen.fd.ping_timeout: 30s 
discovery.zen.fd.ping_retries: 3 
discovery.zen.ping.timeout: 30s 
discovery.zen.ping.multicast.enabled: false 
discovery.zen.minimum_master_nodes: 2

cat/node

node-3.es-pro.com 192.168.254.239 67  99 1.12 d - node-03-239     
node-1.es-pro.com 192.168.254.237 60 100 2.97 d - node-01-237     
node-0.es-pro.com 192.168.254.236  5  60 0.27 - - client-node-236 
node-7.es-pro.com 192.168.250.215 71  95 0.24 d - node-07-215     
node-5.es-pro.com 192.168.254.214 62 100 3.13 d m node-05-214     
node-2.es-pro.com 192.168.254.238 50  97 1.65 d - node-02-238     
node-4.es-pro.com 192.168.254.215 43 100 3.00 d m node-04-215     
node-8.es-pro.com 192.168.250.216 64 100 2.56 d - node-08-216     
node-6.es-pro.com 192.168.254.213 69  99 1.89 d * node-06-213

( 3 num )Master Log:

[2019-06-28 20:36:47,136][WARN ][cluster.action.shard     ] [node-06-213] [ffffffffn_2016][0] received shard failed for target shard [[ffffffffn_2016][0], node[6wfUIeWlTfGmxADxqdBsRw], relocating [yZkxF7GiTVW9CQm4NQBGjA], [P], v[237], s[INITIALIZING], a[id=bHW3WXeZQYmoNsO391yzWQ, rId=g57zPop3TZu5iyhqcwNcVA], expected_shard_size[58285273234]], indexUUID [xJ6n1X8bREWws63uZHBb4g], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:36:57,988][WARN ][transport.netty          ] [node-06-213] exception caught on transport layer [[id: 0x3a951ccb, /192.168.250.216:57213 => /192.168.254.213:6300]], closing connection
java.io.IOException:  connection timeout
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
	at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
	at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
	at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
	at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
	at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
[2019-06-28 20:37:32,482][INFO ][cluster.service          ] [node-06-213] added {{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false},}, reason: zen-disco-join(join from node[{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}])
[2019-06-28 20:39:47,942][INFO ][cluster.routing.allocation] [node-06-213] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ffffffffn_2016][0]] ...]).
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard     ] [node-06-213] [ffffffffn_2016][1] received shard failed for target shard [[ffffffffn_2016][1], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[131], s[STARTED], a[id=Vz_0RHCyS3eC7AV70Xcxwg]], indexUUID [grryTVt0Rr2NY3Jrsq2MHw], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard     ] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard     ] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:27,583][WARN ][action.bulk              ] [node-06-213] [invoice_detail_2015][0] failed to perform indices:data/write/bulk[s][r] on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] 
[2019-06-28 20:42:27,583][WARN ][cluster.action.shard     ] [node-06-213] [invoice_detail_2015][0] received shard failed for target shard [[invoice_detail_2015][0], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[221], s[STARTED], a[id=wxsACNOMR_SGO8i7WTcDMQ]], indexUUID [HRzC8xbsQxuON4lmvH1c3A], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected] 

········

( 8 num )Node Log:

Problem:

I don't know why it's been added and removed then added and deleted again..again .········ ?**

2

This indicates a problem with your network. It's hard to give more specific advice about a version as old as 2.4.6, except to recommend that you upgrade to a more recent version.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.