Elasticsearch version** : 2.4.6
JVM version** :
java version "1.8.0_144"
### OS version** : Centos 64
elastic yml :
····
gateway.recover_after_nodes: 2
gateway.expected_nodes: 2
gateway.recover_after_time: 5m
discovery.zen.fd.ping_interval: 1s
discovery.zen.fd.ping_timeout: 30s
discovery.zen.fd.ping_retries: 3
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.minimum_master_nodes: 2
cat/node
node-3.es-pro.com 192.168.254.239 67 99 1.12 d - node-03-239
node-1.es-pro.com 192.168.254.237 60 100 2.97 d - node-01-237
node-0.es-pro.com 192.168.254.236 5 60 0.27 - - client-node-236
node-7.es-pro.com 192.168.250.215 71 95 0.24 d - node-07-215
node-5.es-pro.com 192.168.254.214 62 100 3.13 d m node-05-214
node-2.es-pro.com 192.168.254.238 50 97 1.65 d - node-02-238
node-4.es-pro.com 192.168.254.215 43 100 3.00 d m node-04-215
node-8.es-pro.com 192.168.250.216 64 100 2.56 d - node-08-216
node-6.es-pro.com 192.168.254.213 69 99 1.89 d * node-06-213
( 3 num )Master Log:
[2019-06-28 20:36:47,136][WARN ][cluster.action.shard ] [node-06-213] [ffffffffn_2016][0] received shard failed for target shard [[ffffffffn_2016][0], node[6wfUIeWlTfGmxADxqdBsRw], relocating [yZkxF7GiTVW9CQm4NQBGjA], [P], v[237], s[INITIALIZING], a[id=bHW3WXeZQYmoNsO391yzWQ, rId=g57zPop3TZu5iyhqcwNcVA], expected_shard_size[58285273234]], indexUUID [xJ6n1X8bREWws63uZHBb4g], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:36:57,988][WARN ][transport.netty ] [node-06-213] exception caught on transport layer [[id: 0x3a951ccb, /192.168.250.216:57213 => /192.168.254.213:6300]], closing connection
java.io.IOException: connection timeout
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
[2019-06-28 20:37:32,482][INFO ][cluster.service ] [node-06-213] added {{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false},}, reason: zen-disco-join(join from node[{node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}])
[2019-06-28 20:39:47,942][INFO ][cluster.routing.allocation] [node-06-213] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[ffffffffn_2016][0]] ...]).
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard ] [node-06-213] [ffffffffn_2016][1] received shard failed for target shard [[ffffffffn_2016][1], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[131], s[STARTED], a[id=Vz_0RHCyS3eC7AV70Xcxwg]], indexUUID [grryTVt0Rr2NY3Jrsq2MHw], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard ] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:24,074][WARN ][cluster.action.shard ] [node-06-213] [ffffffffn_2016][2] received shard failed for target shard [[ffffffffn_2016][2], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[167], s[STARTED], a[id=yTJ3AoDMQzykwwh4A66p3g]], indexUUID [BPii61cNQECZ2nVCHZAdKA], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:27,583][WARN ][action.bulk ] [node-06-213] [invoice_detail_2015][0] failed to perform indices:data/write/bulk[s][r] on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
[2019-06-28 20:42:27,583][WARN ][cluster.action.shard ] [node-06-213] [invoice_detail_2015][0] received shard failed for target shard [[invoice_detail_2015][0], node[6wfUIeWlTfGmxADxqdBsRw], [R], v[221], s[STARTED], a[id=wxsACNOMR_SGO8i7WTcDMQ]], indexUUID [HRzC8xbsQxuON4lmvH1c3A], message [failed to perform indices:data/write/bulk[s] on replica on node {node-08-216}{6wfUIeWlTfGmxADxqdBsRw}{node-8.es-pro.com}{192.168.250.216:6300}{master=false}], failure [NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]]
NodeDisconnectedException[[node-08-216][192.168.250.216:6300][indices:data/write/bulk[s][r]] disconnected]
········
( 8 num )Node Log:
Problem:
I don't know why it's been added and removed then added and deleted again..again .········ ?**