# Our cluster crushed when a node cannot reached

**URL:** <https://discuss.elastic.co/t/our-cluster-crushed-when-a-node-cannot-reached/29921>\
**Category:** Elasticsearch\
**Created:** [September 24, 2015, 2:51pm UTC](https://discuss.elastic.co/t/our-cluster-crushed-when-a-node-cannot-reached/29921 "2015-09-24T14:51:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![chenjinyuan87](https://avatars.discourse-cdn.com/v4/letter/c/bb73d2/32.png) [@chenjinyuan87](https://discuss.elastic.co/u/chenjinyuan87)\
**Post date:** [September 24, 2015, 2:51pm UTC](https://discuss.elastic.co/t/our-cluster-crushed-when-a-node-cannot-reached/29921/1 "2015-09-24T14:51:05Z")

</div>

The only thing we know now is one of our machine may exists some problem on it's network.  
When that happen, all the data nodes in the elasticsearch clusters begin to reconnect it endlessly.  
The logs shows like this:  
[2015-09-23 17:48:01,930][WARN][discovery.zen.ping.multicast] [bak4] failed to connect to requesting node [Ox][ma0VLbs6TfGWrTdSbpfeBQ][TJ-app1][inet[/192.168.2.167:9300]]{client=true, data=false}  
...  
[2015-09-23 17:48:01,930][WARN][transport.netty] [bak4] exception caught on transport layer [[id: 0x95434756]], closing connection  
java.net.NoRouteToHostException: No route to host  
...

And we found thousands of threads waiting  
at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)  
at org.elasticsearch.common.util.concurrent.KeyedLock.acquire(KeyedLock.  
java:64)  
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyT  
ransport.java:649)  
at org.elasticsearch.transport.netty.NettyTransport.connectToNode(NettyT  
ransport.java:630)  
at org.elasticsearch.transport.TransportService.connectToNode(TransportS  
ervice.java:149)  
...

At last, all the data nodes crushed for OOM error, because no more threads can be created.

Our elasticsearch version is 1.3.4.  
Is this a known bug? Or we have done something error causing this problem?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:48pm UTC](https://discuss.elastic.co/t/our-cluster-crushed-when-a-node-cannot-reached/29921/2 "2017-07-05T23:48:15Z")

</div>