# Cluster Failure **URL:** \ **Category:** Elasticsearch\ **Created:** [November 14, 2011, 11:03pm UTC](https://discuss.elastic.co/t/cluster-failure/5872 "2011-11-14T23:03:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [November 14, 2011, 11:03pm UTC](https://discuss.elastic.co/t/cluster-failure/5872/1 "2011-11-14T23:03:05Z")
We saw 100+ cpu usage and then upon reboot we say this: [2011-11-14 15:00:45,785][WARN][indices.cluster] [Network] [documents][10] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [documents][10] shard allocated for local recovery (post api), should exists, but doesn't at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: 99) at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:179) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) [2011-11-14 15:00:45,793][WARN][indices.cluster] [Network] [documents][14] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [documents][14] shard allocated for local recovery (post api), should exists, but doesn't at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: 99) at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:179) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] sending failed shard for [documents][14], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [14] shard allocated for local recovery (post api), should exists, but doesn't]]] [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] received shard failed for [documents][14], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [14] shard allocated for local recovery (post api), should exists, but doesn't]]] [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] sending failed shard for [documents][10], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [10] shard allocated for local recovery (post api), should exists, but doesn't]]] [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] received shard failed for [documents][10], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [10] shard allocated for local recovery (post api), should exists, but doesn't]]] [2011-11-14 15:00:45,795][WARN][cluster.action.shard] [Network] sending failed shard for [documents][18], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [18] failed recovery]; nested: EngineCreationFailureException[[documents][18] Failed to open reader on writer]; nested: FileNotFoundException[/var/lib/elasticsearch/ viralheat/nodes/0/indices/documents/18/index/\_len.fnm (Too many open files)]; ]] [2011-11-14 15:00:45,795][WARN][cluster.action.shard] [Network] received shard failed for [documents][18], node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[documents] [18] failed recovery]; nested: EngineCreationFailureException[[documents][18] Failed to open reader on writer]; nested: FileNotFoundException[/var/lib/elasticsearch/ viralheat/nodes/0/indices/documents/18/index/\_len.fnm (Too many open files)]; ]] [2011-11-14 15:00:45,804][WARN][indices.cluster] [Network] [documents][10] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [documents][10] shard allocated for local recovery (post api), should exists, but doesn't at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: 99) at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:179) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [November 15, 2011, 7:44am UTC](https://discuss.elastic.co/t/cluster-failure/5872/2 "2011-11-15T07:44:55Z")
Please report what you did with more details and the version you used. Also your used environment would be nice (java version, OS, RAM resources, index size, ...) Peter. On 15 Nov., 00:03, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > We saw 100+ cpu usage and then upon reboot we say this: > > [2011-11-14 15:00:45,785][WARN][indices.cluster] [Network] > [documents][10] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [documents][10] shard allocated for local recovery (post api), should > exists, but doesn't > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java: > 99) > at org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:179) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > [2011-11-14 15:00:45,793][WARN][indices.cluster] [Network] > [documents][14] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [documents][14] shard allocated for local recovery (post api), should > exists, but doesn't > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java: > 99) > at org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:179) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] > sending failed shard for [documents][14], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [14] shard allocated for local recovery (post api), should exists, but > doesn't]]] > [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] > received shard failed for [documents][14], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [14] shard allocated for local recovery (post api), should exists, but > doesn't]]] > [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] > sending failed shard for [documents][10], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [10] shard allocated for local recovery (post api), should exists, but > doesn't]]] > [2011-11-14 15:00:45,794][WARN][cluster.action.shard] [Network] > received shard failed for [documents][10], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [10] shard allocated for local recovery (post api), should exists, but > doesn't]]] > [2011-11-14 15:00:45,795][WARN][cluster.action.shard] [Network] > sending failed shard for [documents][18], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [18] failed recovery]; nested: > EngineCreationFailureException[[documents][18] Failed to open reader > on writer]; nested: FileNotFoundException[/var/lib/elasticsearch/ > viralheat/nodes/0/indices/documents/18/index/\_len.fnm (Too many open > files)]; ]] > [2011-11-14 15:00:45,795][WARN][cluster.action.shard] [Network] > received shard failed for [documents][18], > node[UdFv305pR6KwkANc7GdjmQ], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[documents] > [18] failed recovery]; nested: > EngineCreationFailureException[[documents][18] Failed to open reader > on writer]; nested: FileNotFoundException[/var/lib/elasticsearch/ > viralheat/nodes/0/indices/documents/18/index/\_len.fnm (Too many open > files)]; ]] > [2011-11-14 15:00:45,804][WARN][indices.cluster] [Network] > [documents][10] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [documents][10] shard allocated for local recovery (post api), should > exists, but doesn't > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI ndexShardGateway.java: > 99) > at org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:179) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:48am UTC](https://discuss.elastic.co/t/cluster-failure/5872/3 "2017-07-06T03:48:38Z")