# ES nodes crashing: failed to send failed shard **URL:** \ **Category:** Elasticsearch\ **Created:** [October 13, 2016, 6:38am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897 "2016-10-13T06:38:08Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dipathak](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@dipathak](https://discuss.elastic.co/u/dipathak)\ **Post date:** [October 13, 2016, 6:38am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/1 "2016-10-13T06:38:09Z")
Hi, we are running ES 2.0 on a 4 node cluster. ES is crashing 1-5 times in a day on nodes with different reasons. Can someone please suggest why is this happening and what is the way to get out of this situation. Crash is happening on all the 4 nodes every 6-12 hours. After crash cluster works normally, state becomes green, all shards are in STARTED state, until next crash. Pasting one exception here. There are other exceptions also like ShardNotFoundException, java.nio.file.NoSuchFileException, which can be found here [http://pastebin.com/Y0fPULeL](http://pastebin.com/Y0fPULeL). Thanks in advance. [2016-10-06 21:05:10,347][WARN][action.bulk] [130591932414] [cfileindex][5] failed to perform indices:data/write/bulk[s] on node {130591932198}{E4qFPw3-TvizCAeB6ai\_lw}{10.2.34.115}{10.2.34.115:25800}{master=true} TransportException[transport stopped, action: indices:data/write/bulk[s][r]] at org.elasticsearch.transport.TransportService$2.run(TransportService.java:198) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) [2016-10-06 21:05:10,348][WARN][cluster.action.shard] [130591932414] failed to send failed shard to {130591932198}{E4qFPw3-TvizCAeB6ai\_lw}{10.2.34.115}{10.2.34.115:25800}{master=true} SendRequestTransportException[[130591932198][10.2.34.115:25800][internal:cluster/shard/failure]]; nested: TransportException[TransportService is closed stopped can't send request]; at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:323) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:282) at org.elasticsearch.cluster.action.shard.ShardStateAction.innerShardFailed(ShardStateAction.java:98) at org.elasticsearch.cluster.action.shard.ShardStateAction.shardFailed(ShardStateAction.java:88) at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicationPhase$1.handleException(TransportReplicationAction.java:895) at org.elasticsearch.transport.TransportService$2.run(TransportService.java:198) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: TransportException[TransportService is closed stopped can't send request] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:303) ... 8 more [2016-10-06 21:05:10,348][WARN][cluster.action.shard] [130591932414] failed to send failed shard to {130591932198}{E4qFPw3-TvizCAeB6ai\_lw}{10.2.34.115}{10.2.34.115:25800}{master=true} SendRequestTransportException[[130591932198][10.2.34.115:25800][internal:cluster/shard/failure]]; nested: TransportException[TransportService is closed stopped can't send request]; ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 13, 2016, 6:48am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/2 "2016-10-13T06:48:56Z")
Can you please provide some more details about your cluster? How are the nodes configured? How are they deployed? What is the specification of the nodes? How much data? ---
**Author:** ![dipathak](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@dipathak](https://discuss.elastic.co/u/dipathak)\ **Post date:** [October 13, 2016, 9:37am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/3 "2016-10-13T09:37:20Z")
Hi Christian, we have deployed ES by downloading the zip file. Used space by ES data directory on each node is around 50G and there is ample free space. Its a 32 core physical machine, but ours is a shared environment, so other processes are also running on machine. Dedicated memory given for heap is 3G. We have specified settings in elasticsearch.yml file. We have 2 indexes, one with 5 shards and other one with 10. We create indices only once and never delete them. Please let me know if any other information is required. Thanks. ====== elasticsearch.yml file ========= cluster.routing.allocation.disk.watermark.low: 20gb cluster.routing.allocation.disk.watermark.high: 500mb [node.name](http://node.name): 130591932414 node.master: true node.data: true discovery.zen.ping.timeout: 5s discovery.zen.minimum\_master\_nodes: 2 indices.recovery.max\_bytes\_per\_sec: 80mb index.refresh\_interval: 10s index.merge.scheduler.max\_thread\_count: 1 http.port: 25700 transport.tcp.port: 25800 network.bind\_host: 0.0.0.0 network.publish\_host: 10.2.34.119 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [10.2.34.121,10.2.34.115,10.2.34.117] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 13, 2016, 9:44am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/4 "2016-10-13T09:44:00Z")
Can you please provide the output from the [\_cat/nodes API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-nodes.html#cat-nodes)? Do all the nodes have the same configuration? ---
**Author:** ![dipathak](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@dipathak](https://discuss.elastic.co/u/dipathak)\ **Post date:** [October 14, 2016, 5:26am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/5 "2016-10-14T05:26:41Z")
Yes. All the nodes have the same configuration. Thanks ! host ip heap.percent ram.percent load node.role master name 10.2.34.117 10.2.34.117 25 99 27.92 d m 130591932210 10.2.34.119 10.2.34.119 16 99 53.95 d m 130591932414 10.2.34.115 10.2.34.115 15 99 30.35 d \* 130591932198 10.2.34.121 10.2.34.121 22 99 39.68 d m 130591952482 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 14, 2016, 5:31am UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/6 "2016-10-14T05:31:32Z")
> [@dipathak](#): > > discovery.zen.minimum\_master\_nodes: 2 It may have nothing to do with the errors you are seeing, but as you have 4 master eligible nodes, minimum\_master\_nodes should be set to 3 in order to avoid split brain scenarios. I would recommend eliminating this as a potential issue. Is there anything else in the logs around the time the nodes crash? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:12pm UTC](https://discuss.elastic.co/t/es-nodes-crashing-failed-to-send-failed-shard/62897/7 "2017-07-05T22:12:29Z")
Closed on July 5, 2017, 10:12pm UTC