# Failed to start shard **URL:** \ **Category:** Elasticsearch\ **Created:** [September 9, 2012, 1:05am UTC](https://discuss.elastic.co/t/failed-to-start-shard/8962 "2012-09-09T01:05:09Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![shengcer](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@shengcer](https://discuss.elastic.co/u/shengcer)\ **Post date:** [September 9, 2012, 1:05am UTC](https://discuss.elastic.co/t/failed-to-start-shard/8962/1 "2012-09-09T01:05:09Z")
Hi, My elasticsearch instance has been crashed for the 3rd time after the system rebooted and the instance was therefore bounced. I would much appreciate if someone can help me out on this... I have tried to increase the number of replicas and update elasticsearch instance to the newer version 0.19.8, but none of them actually worked. Every time elasticsearch crashed, the pattern was pretty similar ( listed as follows). It will keep rolling suggesting almost all indices have this issue and create a log of monster size. For your information, I indexed 30 indices with size around ~25GB to a elasticsearch cluster with 2 nodes, 5 shards each, and 2 replicas each. WARNING: [dev-bry200163108d] [coverage-elastic1346994255418][0] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [coverage-elastic1346994255418][0] failed recovery at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: [coverage-elastic1346994255418][0] Failed to open reader on writer at org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:286) at org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:579) at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:188) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) ... 3 more I do notice that if I disable flush on translog (index._translog_.\* disable\_flush=true\*), the cluster would be fine even it is killed due to system reboot. If I can guarantee a flush operation is executed for all writing/updating index operations, can I safely just disable the translog flush forever? -- ---
**Author:** ![shengcer](https://avatars.discourse-cdn.com/v4/letter/s/41988e/32.png) [@shengcer](https://discuss.elastic.co/u/shengcer)\ **Post date:** [September 9, 2012, 1:22am UTC](https://discuss.elastic.co/t/failed-to-start-shard/8962/2 "2012-09-09T01:22:24Z")
I forgot to mention that the hard limit on OS is 16384, and I wrote a tool to start elasticsearch always with that limit. When OS reboots, that tool would be automatically called to bring up elasticsearch. But I still get too many open files open error, and log shows some files are missing WARNING: [dev-bry200163111d] sending failed shard for [client-elastic1347095014701][1], node[coS8PH1jQdq78FCkjY\_TLw], [R], s[INITIALIZING], reason [Failed to start shard, message [RecoveryFailedException[[client-elastic1347095014701][1]: Recovery failed from [dev-bry200163108d][qqCnD112Qiqu8qLWxelo1w][inet[/171.150.210.106:14701]] into [dev-bry200163111d][coS8PH1jQdq78FCkjY\_TLw][inet[/171.150.210.107:14701]]]; nested: RemoteTransportException[[dev-bry200163108d][inet[/171.150.210.106:14701]][index/shard/recovery/startRecovery]]; nested: RecoveryEngineException[[client-elastic1347095014701][1] Phase[1] Execution failed]; nested: RecoverFilesRecoveryException[[client-elastic1347095014701][1] Failed to transfer [1] files with total size of [147.9mb]]; nested: FileNotFoundException[/.statelite/tmpfs/data/clobber/local0/services/dev/elasticsearch/elasticsearch-0.0.0/cache/clobber-dev/nodes/0/indices/client-elastic1347095014701/1/index/segments\_m (Too many open files)]; ]] This only happens when the cluster crashes in a disastrous manner, so I don't really buy that raising hard limit of OS would be the fix. On Saturday, September 8, 2012 9:05:09 PM UTC-4, Sheng wrote: > Hi, > > My elasticsearch instance has been crashed for the 3rd time after the > system rebooted and the instance was therefore bounced. I would much > appreciate if someone can help me out on this... > > I have tried to increase the number of replicas and update elasticsearch > instance to the newer version 0.19.8, but none of them actually worked. > Every time elasticsearch crashed, the pattern was pretty similar ( listed > as follows). It will keep rolling suggesting almost all indices have this > issue and create a log of monster size. For your information, I indexed 30 > indices with size around ~25GB to a elasticsearch cluster with 2 nodes, 5 > shards each, and 2 replicas each. > > WARNING: [dev-bry200163108d] [coverage-elastic1346994255418][0] failed to > start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [coverage-elastic1346994255418][0] failed recovery > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.elasticsearch.index.engine.EngineCreationFailureException: > [coverage-elastic1346994255418][0] Failed to open reader on writer > at > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:286) > at > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:579) > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:188) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > ... 3 more > > I do notice that if I disable flush on translog (index._translog_.\* > disable\_flush=true\*), the cluster would be fine even it is killed due to > system reboot. If I can guarantee a flush operation is executed for all > writing/updating index operations, can I safely just disable the translog > flush forever? -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:13am UTC](https://discuss.elastic.co/t/failed-to-start-shard/8962/3 "2017-07-06T03:13:34Z")