# How to restart/recover a shard?

**URL:** <https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239>\
**Category:** Elasticsearch\
**Created:** [August 19, 2010, 11:28am UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239 "2010-08-19T11:28:33Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 5:30pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/21 "2010-08-20T17:30:51Z")

</div>

Probably need some better error detection... 😉

Thanks for the help.

...Ken

Shay Banon wrote:

> It means that the gateway store got corrupted. You will have to rebuild  
> the index. Probably due to all HEAD changes... . Hopefully its getting  
> stable now.
> 
> -shay.banon
> 
> On Fri, Aug 20, 2010 at 8:13 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> Looks like a file may be missing on the gateway... this repeats in the
> log over and over.
> 
> [12:10:00,597][WARN][indices.cluster] [Magilla] [twitter][1]
> failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> [twitter][1] Failed to recover translog
> at
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:516)
> at
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:417)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:172)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:636)
> Caused by:
> org.elasticsearch.index.engine.EngineCreationFailureException:
> [twitter][1] Failed to open reader on writer
> at
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:405)
> at
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:440)
> ... 5 more
> Caused by: java.io.FileNotFoundException:
> /mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/_d8g.cfs
> (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<init>(SimpleFSDirectory.java:76)
> at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleFSDirectory.java:97)
> at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirectory.java:87)
> at
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)
> at
> org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInput(AbstractStore.java:287)
> at
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:67)
> at
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java:114)
> at
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)
> at
> org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:574)
> at
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:150)
> at
> org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryReader.java:36)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
> at
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)
> at
> org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngine.java:538)
> at
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158)
> ... 7 more
> [12:10:00,605][WARN][cluster.action.shard] [Magilla] sending
> failed shard for [twitter][1],
> node[10dab323-019b-4036-854f-89bb068dcc8d], [P], s[INITIALIZING], reason
> [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[twitter][1] Failed to recover
> translog]; nested: EngineCreationFailureException[[twitter][1] Failed to
> open reader on writer]; nested:
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/_d8g.cfs
> (No such file or directory)]; ]]
> 
> Shay Banon wrote:
> > Also, use the latest again, pushed some more fixes.
> >
> > On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon
> > <shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>> wrote:
> >
> > Do you see any exceptions in the logs (failing to start the
> shard)?
> >
> >
> > On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > Now its looping: progress is going to 100, then starting
> over.
> >
> > I set up a 1/second loop using:
> > while /bin/true; do date; curl -XGET
> > 'http://192.168.1.5:9200/twitter/_status?pretty=true';
> sleep 1; done
> > then copied it to gist at: http://gist.github.com/540711
> >
> > It should have recovered by now, I would think.
> >
> > ...Ken
> >
> > Shay Banon wrote:
> > > great, ping me if it does not end, I am here to help (we can
> > make it
> > > more interactive on IRC).
> > >
> > > p.s. Can you keep the original json format when you
> gist? Much
> > easier to
> > > know whats going on. You can add pretty=true as a
> parameter to
> > get it
> > > pretty printed.
> > >
> > > -shay.banon
> > >
> > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >
> > > I think so... Here's the latest on gist
> > http://gist.github.com/540471
> > >
> > > Thanks for the pointer on gist, I've never used it
> before.
> > >
> > > Shay Banon wrote:
> > > > The top just states which shards were queries, a shard
> > that is
> > > still not
> > > > allocated will obviously not be allocated. It
> seems like
> > its still in
> > > > recovery process. There are two main APIs to really
> > understand what is
> > > > going on (except for the high level health api), the
> > cluster state
> > > API,
> > > > that shows you what the cluster wide state is (where
> > each shard is
> > > > supposed to be, what its state is), and the status api
> > which gives you
> > > > detailed information of the status of each shard
> > allocated on each
> > > node.
> > > >
> > > > Is the recovery progressing?
> > > >
> > > > p.s. Can you use gist instead of pastebin?
> > > >
> > > > -shay.banon
> > > >
> > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > >
> > > > I restarted and now 35 of 36 are successful,
> but if
> > you look
> > > at the
> > > > status, it's showing multiple shards in recovery.
> > I'm confused.
> > > >
> > > > See cluster status in http://pastebin.com/9qWLf3mk
> > > >
> > > > Kenneth Loafman wrote:
> > > > > Will do so in just a bit...
> > > > >
> > > > > Shay Banon wrote:
> > > > >> ... can you test?
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > > >> <shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>
> > > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>>
> > > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>
> > > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>>>> wrote:
> > > > >>
> > > > >> Just pushed a fix for this.
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 3:31 PM,
> Kenneth Loafman
> > > > >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > > > >>
> > > > >> Attachments did not make it. See:
> > > > >> http://pastebin.com/ziALRgx5 --
> cluster state
> > > > >> http://pastebin.com/63Xm95xM --
> index status
> > > > >>
> > > > >> Sorry, they lost their formatting
> on Pastbin.
> > > > >>
> > > > >> ...Ken
> > > > >>
> > > > >> Kenneth Loafman wrote:
> > > > >> > I upgraded to last nights version,
> > restarted, and
> > > > things are
> > > > >> worse. Now
> > > > >> > I have 5 shards hung at recover, not
> > all on the same
> > > > node. Weird.
> > > > >> >
> > > > >> > I've attached the info you want.
> I'll
> > leave
> > > things running
> > > > >> for now.
> > > > >> >
> > > > >> > ...Thanks,
> > > > >> > ...Ken
> > > > >> >
> > > > >> > Shay Banon wrote:
> > > > >> >> Do you still have it running?
> Can you
> > gist the
> > > cluster
> > > > state
> > > > >> and the
> > > > >> >> index status results?
> > > > >> >>
> > > > >> >> I see that you are using master, I
> > have fixed
> > > several
> > > > things
> > > > >> in this
> > > > >> >> area, can you pull a new version?
> > > > >> >>
> > > > >> >> -shay.banon
> > > > >> >>
> > > > >> >> On Fri, Aug 20, 2010 at 12:33 AM,
> > Kenneth Loafman
> > > > >> >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > > > >> >>
> > > > >> >> It seems to have started
> recover,
> > but it's been
> > > > 7.5 hours
> > > > >> and appears to
> > > > >> >> be stopped/hung...
> > > > >> >>
> > > > >> >> > "1": [
> > > > >> >> > {
> > > > >> >> >
> > "gateway_recovery": {
> > > > >> >> >
> > "index": {
> > > > >> >> >
> > > > >> "expected_recovered_size": "0b",
> > > > >> >> >
> > > > >> "expected_recovered_size_in_bytes": 0,
> > > > >> >> >
> > > > "recovered_size": "0b",
> > > > >> >> >
> > > > >> "recovered_size_in_bytes": 0,
> > > > >> >> >
> > > "reused_size": "0b",
> > > > >> >> >
> > > > "reused_size_in_bytes": 0,
> > > > >> >> >
> > "size": "0b",
> > > > >> >> >
> > > "size_in_bytes": 0,
> > > > >> >> >
> > > > "throttling_time": "0s",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 0
> > > > >> >> > },
> > > > >> >> >
> > "stage": "RETRY",
> > > > >> >> >
> > > "start_time_in_millis":
> > > > >> 1282226019603,
> > > > >> >> >
> > "throttling_time":
> > > > "7.6h",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 27514627,
> > > > >> >> >
> > "time": "7.6h",
> > > > >> >> >
> > "time_in_millis":
> > > > 27514657,
> > > > >> >> >
> > "translog": {
> > > > >> >> >
> > "recovered": 0
> > > > >> >> > }
> > > > >> >> > },
> > > > >> >> >                        
> "index": {
> > > > >> >> >
> > "size": "0b",
> > > > >> >> >
> > "size_in_bytes": 0
> > > > >> >> > },
> > > > >> >> >                        
> "routing": {
> > > > >> >> >                          
> "index":
> > > "twitter",
> > > > >> >> >                          
> "node":
> > > > >> >>
> > "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > > >> >> >
> > "primary": true,
> > > > >> >> >
> > > "relocating_node": null,
> > > > >> >> >
> > "shard": 1,
> > > > >> >> >                          
> "state":
> > > "INITIALIZING"
> > > > >> >> > },
> > > > >> >> >                        
> "state":
> > "RECOVERING"
> > > > >> >> > }
> > > > >> >> > ]
> > > > >> >>
> > > > >> >>
> > > > >> >> Shay Banon wrote:
> > > > >> >> > It should be allocated on the
> > other node, you
> > > > shouldn't
> > > > >> need to start
> > > > >> >> > another node. When you issue a
> > cluster health
> > > > (simple
> > > > >> curl can
> > > > >> >> do), what
> > > > >> >> > is the status? The cluster
> state
> > API gives
> > > you more
> > > > >> information if you
> > > > >> >> > are after (each shard and its
> > state).
> > > > >> >> >
> > > > >> >> > On Thu, Aug 19, 2010 at
> 3:48 PM,
> > Kenneth
> > > Loafman
> > > > >> >> > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>>> wrote:
> > > > >> >> >
> > > > >> >> > No this is the first
> time. The
> > > shutdown took a
> > > > >> while with several
> > > > >> >> > 'Waiting for not to
> > shutdown..." style
> > > > message. It
> > > > >> came up bad
> > > > >> >> > after that.
> > > > >> >> >
> > > > >> >> > So, if I have two
> nodes now,
> > and one
> > > needs to be
> > > > >> recovered,
> > > > >> >> I'll need 3
> > > > >> >> > nodes to get the
> recovery done?
> > > > >> >> >
> > > > >> >> > ...Ken
> > > > >> >> >
> > > > >> >> > Shay Banon wrote:
> > > > >> >> > > The shard will allocated
> > to another
> > > node and
> > > > >> recovered there. Do
> > > > >> >> > you see
> > > > >> >> > > it happen continuously?
> > > > >> >> > >
> > > > >> >> > > -shay.banon
> > > > >> >> > >
> > > > >> >> > > On Thu, Aug 19, 2010 at
> > 2:28 PM, Kenneth
> > > > Loafman
> > > > >> >> > >
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>>
> > > > >> >> >
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>
> > > > >> >> >
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >>    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>>>> wrote:
> > > > >> >> > >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > > The second shard on
> > one of my
> > > indexes has
> > > > >> failed due to:
> > > > >> >> > > [05:59:47,332][WARN
> > ][index.gateway
> > > > >> ] [Mangog]
> > > > >> >> > [twitter][1]
> > > > >> >> > > failed to
> snapshot on
> > close
> > > > >> >> > > ...followed by a
> long
> > traceback.
> > > > >> >> > > ...followed by:
> > > > >> >> > > [05:59:49,336][WARN
> > > ][cluster.action.shard
> > > > >> ] [Mangog]
> > > > >> >> > received shard
> > > > >> >> > > failed for
> [twitter][1],
> > > > >> >> >
> > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > >> >> > > [P],
> s[INITIALIZING],
> > reason [Failed
> > > > to start
> > > > >> shard, message
> > > > >> >> > >
> > > > >>
> > [IndexShardGatewayRecoveryException[[twitter][1]
> > > Failed to
> > > > >> >> > recovery
> > > > >> >> > > translog]; nested:
> > > > >> >>
> > EngineCreationFailureException[[twitter][1]
> > > > >> >> > Failed to
> > > > >> >> > > open reader on
> > writer]; nested:
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> >        
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > > > >> >> > > (No such file or
> > directory)]; ]]
> > > > >> >> > >
> > > > >> >> > > Is the recovery
> process
> > > automatic, or do I
> > > > >> have to do
> > > > >> >> something
> > > > >> >> > > special? It appears
> > to be just this
> > > > one shard.
> > > > >> >> > >
> > > > >> >> > > I use the service
> > wrapper to
> > > start/stop
> > > > >> 0.9.1-SNAPSHOT,
> > > > >> >> and my
> > > > >> >> > config is
> > > > >> >> > > below.
> > > > >> >> > >
> > > > >> >> > > ...Thanks,
> > > > >> >> > > ...Ken
> > > > >> >> > >
> > > > >> >> > > cloud:
> > > > >> >> > > aws:
> > > > >> >> > >          
> access_key: *****
> > > > >> >> > >          
> secret_key: *****
> > > > >> >> > >
> > > > >> >> > > gateway:
> > > > >> >> > > type: s3
> > > > >> >> > > s3:
> > > > >> >> > > bucket: *****
> > > > >> >> > >
> > > > >> >> > > path :
> > > > >> >> > > work :
> > /mnt/search-data-dev
> > > > >> >> > > logs :
> > > /mnt/search-data-dev/node1/logs
> > > > >> >> > >
> > > > >> >> > > index :
> > > > >> >> > >      
> number_of_shards : 2
> > > > >> >> > >      
> number_of_replicas : 1
> > > > >> >> > >
> > > > >> >> > > network :
> > > > >> >> > > host :
> 192.168.1.5
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> >
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![Grant\_Rodgers](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grant_rodgers/32/3204_2.png) [@Grant\_Rodgers](https://discuss.elastic.co/u/Grant_Rodgers)\
**Post date:** [August 20, 2010, 6:19pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/22 "2010-08-20T18:19:02Z")

</div>

FWIW, I've been having similar issues with my cluster in the past few  
days. Some shards get stuck recovering and never make progress.

I have tried several different HEADs, and the first one that worked  
was b785979. I deleted all the work directories and restarted the  
cluster, and it recovered all primary shards in about 30 minutes, and  
all replicas about 20 minutes later. This is about 90 gigs worth of  
index recovered in less than an hour. Fortunately our gateway is not  
corrupted.

On Aug 20, 10:30 am, Kenneth Loafman [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)  
wrote:

> Probably need some better error detection... 😉
> 
> Thanks for the help.
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > It means that the gateway store got corrupted. You will have to rebuild  
> > the index. Probably due to all HEAD changes... . Hopefully its getting  
> > stable now.
> 
> > -shay.banon
> 
> > On Fri, Aug 20, 2010 at 8:13 PM, Kenneth Loafman  
> > \<[kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com) [mailto:kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)\> wrote:
> 
> > ```
> > Looks like a file may be missing on the gateway... this repeats in the
> > log over and over.
> > 
> > ```
> 
> > ```
> > [12:10:00,597][WARN][indices.cluster] [Magilla] [twitter][1]
> > failed to start shard
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> > [twitter][1] Failed to recover translog
> > at
> > org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove rTranslog(BlobStoreIndexShardGateway.java:516)
> > at
> > org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove r(BlobStoreIndexShardGateway.java:417)
> > at
> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa tewayService.java:172)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 110)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 603)
> > at java.lang.Thread.run(Thread.java:636)
> > Caused by:
> > org.elasticsearch.index.engine.EngineCreationFailureException:
> > [twitter][1] Failed to open reader on writer
> > at
> > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171 )
> > at
> > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPre pareForTranslog(InternalIndexShard.java:405)
> > at
> > org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove rTranslog(BlobStoreIndexShardGateway.java:440)
> > ... 5 more
> > Caused by: java.io.FileNotFoundException:
> > /mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/_d8g.cfs
> > (No such file or directory)
> > at java.io.RandomAccessFile.open(Native Method)
> > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> > at
> > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<in it>(SimpleFSDirectory.java:76)
> > at
> > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleF SDirectory.java:97)
> > at
> > org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirector y.java:87)
> > at
> > org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)
> > at
> > org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInpu t(AbstractStore.java:287)
> > at
> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:6 7)
> > at
> > org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java :114)
> > at
> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)
> > at
> > org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter .java:574)
> > at
> > org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:150)
> > at
> > org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRea der.java:36)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)
> > at
> > org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngi ne.java:538)
> > at
> > org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158 )
> > ... 7 more
> > [12:10:00,605][WARN][cluster.action.shard] [Magilla] sending
> > failed shard for [twitter][1],
> > node[10dab323-019b-4036-854f-89bb068dcc8d], [P], s[INITIALIZING], reason
> > [Failed to start shard, message
> > [IndexShardGatewayRecoveryException[[twitter][1] Failed to recover
> > translog]; nested: EngineCreationFailureException[[twitter][1] Failed to
> > open reader on writer]; nested:
> > FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/tw itter/1/index/_d8g.cfs
> > (No such file or directory)]; ]]
> > 
> > ```
> 
> > ```
> > Shay Banon wrote:
> > > Also, use the latest again, pushed some more fixes.
> > 
> > ```
> 
> > ```
> > > On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon
> > > <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > 
> > ```
> 
> > ```
> > > Do you see any exceptions in the logs (failing to start the
> > shard)?
> > 
> > ```
> 
> > ```
> > > On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> > > <kenneth.loaf...@gmail.com <mailto:kenneth.loaf...@gmail.com>
> > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>> wrote:
> > 
> > ```
> 
> > ```
> > > Now its looping: progress is going to 100, then starting
> > over.
> > 
> > ```
> 
> > ```
> > > I set up a 1/second loop using:
> > > while /bin/true; do date; curl -XGET
> > > 'http://192.168.1.5:9200/twitter/_status?pretty=true';
> > sleep 1; done
> > > then copied it to gist at:http://gist.github.com/540711
> > 
> > ```
> 
> > ```
> > > It should have recovered by now, I would think.
> > 
> > ```
> 
> > ```
> > > ...Ken
> > 
> > ```
> 
> > ```
> > > Shay Banon wrote:
> > > > great, ping me if it does not end, I am here to help (we can
> > > make it
> > > > more interactive on IRC).
> > 
> > ```
> 
> > ```
> > > > p.s. Can you keep the original json format when you
> > gist? Much
> > > easier to
> > > > know whats going on. You can add pretty=true as a
> > parameter to
> > > get it
> > > > pretty printed.
> > 
> > ```
> 
> > ```
> > > > -shay.banon
> > 
> > ```
> 
> > ```
> > > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > > > <kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com> <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>>> wrote:
> > 
> > ```
> 
> > ```
> > > > I think so... Here's the latest on gist
> > > http://gist.github.com/540471
> > 
> > ```
> 
> > ```
> > > > Thanks for the pointer on gist, I've never used it
> > before.
> > 
> > ```
> 
> > ```
> > > > Shay Banon wrote:
> > > > > The top just states which shards were queries, a shard
> > > that is
> > > > still not
> > > > > allocated will obviously not be allocated. It
> > seems like
> > > its still in
> > > > > recovery process. There are two main APIs to really
> > > understand what is
> > > > > going on (except for the high level health api), the
> > > cluster state
> > > > API,
> > > > > that shows you what the cluster wide state is (where
> > > each shard is
> > > > > supposed to be, what its state is), and the status api
> > > which gives you
> > > > > detailed information of the status of each shard
> > > allocated on each
> > > > node.
> > 
> > ```
> 
> > ```
> > > > > Is the recovery progressing?
> > 
> > ```
> 
> > ```
> > > > > p.s. Can you use gist instead of pastebin?
> > 
> > ```
> 
> > ```
> > > > > -shay.banon
> > 
> > ```
> 
> > ```
> > > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > > > <kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>>
> > > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> > <mailto:kenneth.loaf...@gmail.com>>>>> wrote:
> > 
> > ```
> 
> > ```
> > > > > I restarted and now 35 of 36 are successful,
> > but if
> > > you look
> > > > at the
> > > > > status, it's showing multiple shards in recovery.
> > > I'm confused.
> > 
> > ```
> 
> > ```
> > > > > See cluster status inhttp://pastebin.com/9qWLf3mk
> > 
> > ```
> 
> > ```
> > > > > Kenneth Loafman wrote:
> > > > > > Will do so in just a bit...
> > 
> > ```
> 
> > ```
> > > > > > Shay Banon wrote:
> > > > > >> ... can you test?
> > 
> > ```
> 
> > ```
> > > > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > > > >> <shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>
> > > > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > <mailto:shay.ba...@elasticsearch.com...
> > 
> > ```
> 
> read more »

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 6:58pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/23 "2010-08-20T18:58:31Z")

</div>

master has not been that stable for the past week or two, hopefully it has  
been resolved now...

On Fri, Aug 20, 2010 at 9:19 PM, Grant Rodgers [grantr@gmail.com](mailto:grantr@gmail.com) wrote:

> FWIW, I've been having similar issues with my cluster in the past few  
> days. Some shards get stuck recovering and never make progress.
> 
> I have tried several different HEADs, and the first one that worked  
> was b785979. I deleted all the work directories and restarted the  
> cluster, and it recovered all primary shards in about 30 minutes, and  
> all replicas about 20 minutes later. This is about 90 gigs worth of  
> index recovered in less than an hour. Fortunately our gateway is not  
> corrupted.
> 
> On Aug 20, 10:30 am, Kenneth Loafman [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)  
> wrote:
> 
> > Probably need some better error detection... 😉
> > 
> > Thanks for the help.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > 
> > > It means that the gateway store got corrupted. You will have to rebuild  
> > > the index. Probably due to all HEAD changes... . Hopefully its getting  
> > > stable now.
> > 
> > > -shay.banon
> > 
> > > On Fri, Aug 20, 2010 at 8:13 PM, Kenneth Loafman  
> > > \<[kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com) [mailto:kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)\> wrote:
> > 
> > > ```
> > > Looks like a file may be missing on the gateway... this repeats in
> > > 
> > > ```
> 
> the
> 
> > > ```
> > > log over and over.
> > > 
> > > ```
> > 
> > > ```
> > > [12:10:00,597][WARN][indices.cluster] [Magilla]
> > > 
> > > ```
> 
> [twitter][1]
> 
> > > ```
> > > failed to start shard
> > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> > > [twitter][1] Failed to recover translog
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove  
> rTranslog(BlobStoreIndexShardGateway.java:516)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove  
> r(BlobStoreIndexShardGateway.java:417)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa  
> tewayService.java:172)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1  
> 110)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:  
> 603)
> 
> > > ```
> > > at java.lang.Thread.run(Thread.java:636)
> > > Caused by:
> > > org.elasticsearch.index.engine.EngineCreationFailureException:
> > > [twitter][1] Failed to open reader on writer
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171  
> )
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPre  
> pareForTranslog(InternalIndexShard.java:405)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove  
> rTranslog(BlobStoreIndexShardGateway.java:440)
> 
> > > ```
> > > ... 5 more
> > > Caused by: java.io.FileNotFoundException:
> > > 
> > > ```
> 
> /mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/\_d8g.cfs
> 
> > > ```
> > > (No such file or directory)
> > > at java.io.RandomAccessFile.open(Native Method)
> > > at
> > > 
> > > ```
> 
> java.io.RandomAccessFile.(RandomAccessFile.java:233)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:76)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleF  
> SDirectory.java:97)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirector  
> y.java:87)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInpu  
> t(AbstractStore.java:287)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:6  
> 7)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.SegmentReader$CoreReaders.(SegmentReader.java  
> :114)
> 
> > > ```
> > > at
> > > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter  
> .java:574)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:150)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryRea  
> der.java:36)
> 
> > > ```
> > > at
> > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
> > > at
> > > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngi  
> ne.java:538)
> 
> > > ```
> > > at
> > > 
> > > ```
> 
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158  
> )
> 
> > > ```
> > > ... 7 more
> > > [12:10:00,605][WARN][cluster.action.shard] [Magilla] sending
> > > failed shard for [twitter][1],
> > > node[10dab323-019b-4036-854f-89bb068dcc8d], [P], s[INITIALIZING],
> > > 
> > > ```
> 
> reason
> 
> > > ```
> > > [Failed to start shard, message
> > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to recover
> > > translog]; nested: EngineCreationFailureException[[twitter][1]
> > > 
> > > ```
> 
> Failed to
> 
> > > ```
> > > open reader on writer]; nested:
> > > 
> > > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/tw  
> itter/1/index/\_d8g.cfs
> 
> > > ```
> > > (No such file or directory)]; ]]
> > > 
> > > ```
> > 
> > > ```
> > > Shay Banon wrote:
> > > > Also, use the latest again, pushed some more fixes.
> > > 
> > > ```
> > 
> > > ```
> > > > On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon
> > > > <shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>> wrote:
> > > 
> > > ```
> > 
> > > ```
> > > > Do you see any exceptions in the logs (failing to start the
> > > shard)?
> > > 
> > > ```
> > 
> > > ```
> > > > On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> > > > <kenneth.loaf...@gmail.com <mailto:kenneth.loaf...@gmail.com
> > > 
> > > ```
> > 
> > > ```
> > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>> wrote:
> > > 
> > > ```
> > 
> > > ```
> > > > Now its looping: progress is going to 100, then starting
> > > over.
> > > 
> > > ```
> > 
> > > ```
> > > > I set up a 1/second loop using:
> > > > while /bin/true; do date; curl -XGET
> > > > 'http://192.168.1.5:9200/twitter/_status?pretty=true';
> > > sleep 1; done
> > > > then copied it to gist at:http://gist.github.com/540711
> > > 
> > > ```
> > 
> > > ```
> > > > It should have recovered by now, I would think.
> > > 
> > > ```
> > 
> > > ```
> > > > ...Ken
> > > 
> > > ```
> > 
> > > ```
> > > > Shay Banon wrote:
> > > > > great, ping me if it does not end, I am here to help
> > > 
> > > ```
> 
> (we can
> 
> > > ```
> > > > make it
> > > > > more interactive on IRC).
> > > 
> > > ```
> > 
> > > ```
> > > > > p.s. Can you keep the original json format when you
> > > gist? Much
> > > > easier to
> > > > > know whats going on. You can add pretty=true as a
> > > parameter to
> > > > get it
> > > > > pretty printed.
> > > 
> > > ```
> > 
> > > ```
> > > > > -shay.banon
> > > 
> > > ```
> > 
> > > ```
> > > > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > > > > <kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com> <mailto:
> > > 
> > > ```
> 
> [kenneth.loaf...@gmail.com](mailto:kenneth.loaf...@gmail.com)
> 
> > > ```
> > > <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>>> wrote:
> > > 
> > > ```
> > 
> > > ```
> > > > > I think so... Here's the latest on gist
> > > > http://gist.github.com/540471
> > > 
> > > ```
> > 
> > > ```
> > > > > Thanks for the pointer on gist, I've never used it
> > > before.
> > > 
> > > ```
> > 
> > > ```
> > > > > Shay Banon wrote:
> > > > > > The top just states which shards were queries, a
> > > 
> > > ```
> 
> shard
> 
> > > ```
> > > > that is
> > > > > still not
> > > > > > allocated will obviously not be allocated. It
> > > seems like
> > > > its still in
> > > > > > recovery process. There are two main APIs to
> > > 
> > > ```
> 
> really
> 
> > > ```
> > > > understand what is
> > > > > > going on (except for the high level health api),
> > > 
> > > ```
> 
> the
> 
> > > ```
> > > > cluster state
> > > > > API,
> > > > > > that shows you what the cluster wide state is
> > > 
> > > ```
> 
> (where
> 
> > > ```
> > > > each shard is
> > > > > > supposed to be, what its state is), and the
> > > 
> > > ```
> 
> status api
> 
> > > ```
> > > > which gives you
> > > > > > detailed information of the status of each shard
> > > > allocated on each
> > > > > node.
> > > 
> > > ```
> > 
> > > ```
> > > > > > Is the recovery progressing?
> > > 
> > > ```
> > 
> > > ```
> > > > > > p.s. Can you use gist instead of pastebin?
> > > 
> > > ```
> > 
> > > ```
> > > > > > -shay.banon
> > > 
> > > ```
> > 
> > > ```
> > > > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > > > > <kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>>
> > > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>
> > > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>
> > > > <mailto:kenneth.loaf...@gmail.com
> > > <mailto:kenneth.loaf...@gmail.com>>>>> wrote:
> > > 
> > > ```
> > 
> > > ```
> > > > > > I restarted and now 35 of 36 are successful,
> > > but if
> > > > you look
> > > > > at the
> > > > > > status, it's showing multiple shards in
> > > 
> > > ```
> 
> recovery.
> 
> > > ```
> > > > I'm confused.
> > > 
> > > ```
> > 
> > > ```
> > > > > > See cluster status inhttp://
> > > 
> > > ```
> 
> [pastebin.com/9qWLf3mk](http://pastebin.com/9qWLf3mk)
> 
> > > ```
> > > > > > Kenneth Loafman wrote:
> > > > > > > Will do so in just a bit...
> > > 
> > > ```
> > 
> > > ```
> > > > > > > Shay Banon wrote:
> > > > > > >> ... can you test?
> > > 
> > > ```
> > 
> > > ```
> > > > > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay
> > > 
> > > ```
> 
> Banon
> 
> > > ```
> > > > > > >> <shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>
> > > > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>
> > > > <mailto:shay.ba...@elasticsearch.com
> > > <mailto:shay.ba...@elasticsearch.com>>>
> > > > > > <mailto:shay.ba...@elasticsearch.com...
> > > 
> > > ```
> > 
> > read more »

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 7:54pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/24 "2010-08-20T19:54:54Z")

</div>

Shay,

I just want to say thanks for all the hard work.

This is an awesome package!

...Ken

Shay Banon wrote:

> master has not been that stable for the past week or two, hopefully it  
> has been resolved now...
> 
> On Fri, Aug 20, 2010 at 9:19 PM, Grant Rodgers \<[grantr@gmail.com](mailto:grantr@gmail.com)  
> [mailto:grantr@gmail.com](mailto:grantr@gmail.com)\> wrote:
> 
> ```
> FWIW, I've been having similar issues with my cluster in the past few
> days. Some shards get stuck recovering and never make progress.
> 
> I have tried several different HEADs, and the first one that worked
> was b785979. I deleted all the work directories and restarted the
> cluster, and it recovered all primary shards in about 30 minutes, and
> all replicas about 20 minutes later. This is about 90 gigs worth of
> index recovered in less than an hour. Fortunately our gateway is not
> corrupted.
> 
> On Aug 20, 10:30 am, Kenneth Loafman <kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> wrote:
> > Probably need some better error detection... ;-)
> >
> > Thanks for the help.
> >
> > ...Ken
> >
> >
> >
> > Shay Banon wrote:
> > > It means that the gateway store got corrupted. You will have to
> rebuild
> > > the index. Probably due to all HEAD changes... . Hopefully its
> getting
> > > stable now.
> >
> > > -shay.banon
> >
> > > On Fri, Aug 20, 2010 at 8:13 PM, Kenneth Loafman
> > > <kenneth.loaf...@gmail.com <mailto:kenneth.loaf...@gmail.com>
> <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>> wrote:
> >
> > > Looks like a file may be missing on the gateway... this
> repeats in the
> > > log over and over.
> >
> > > [12:10:00,597][WARN][indices.cluster] [Magilla]
> [twitter][1]
> > > failed to start shard
> > >    
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> > > [twitter][1] Failed to recover translog
> > > at
> > >    
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove
> rTranslog(BlobStoreIndexShardGateway.java:516)
> > > at
> > >    
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove
> r(BlobStoreIndexShardGateway.java:417)
> > > at
> > >    
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGa
> tewayService.java:172)
> > > at
> > >    
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
> 110)
> > > at
> > >    
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
> 603)
> > > at java.lang.Thread.run(Thread.java:636)
> > > Caused by:
> > > org.elasticsearch.index.engine.EngineCreationFailureException:
> > > [twitter][1] Failed to open reader on writer
> > > at
> > >    
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171
> )
> > > at
> > >    
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPre
> pareForTranslog(InternalIndexShard.java:405)
> > > at
> > >    
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recove
> rTranslog(BlobStoreIndexShardGateway.java:440)
> > > ... 5 more
> > > Caused by: java.io.FileNotFoundException:
> > >    
> /mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/_d8g.cfs
> > > (No such file or directory)
> > > at java.io.RandomAccessFile.open(Native Method)
> > > at
> java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> > > at
> > >    
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.<in
> it>(SimpleFSDirectory.java:76)
> > > at
> > >    
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.<init>(SimpleF
> SDirectory.java:97)
> > > at
> > >    
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.<init>(NIOFSDirector
> y.java:87)
> > > at
> > >    
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)
> > > at
> > >    
> org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInpu
> t(AbstractStore.java:287)
> > > at
> > >    
> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:6
> 7)
> > > at
> > >    
> org.apache.lucene.index.SegmentReader$CoreReaders.<init>(SegmentReader.java
> :114)
> > > at
> > >    
> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)
> > > at
> > >    
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)
> > > at
> > >    
> org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter
> .java:574)
> > > at
> > >    
> org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:150)
> > > at
> > >    
> org.apache.lucene.index.ReadOnlyDirectoryReader.<init>(ReadOnlyDirectoryRea
> der.java:36)
> > > at
> > >    
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)
> > > at
> > >    
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)
> > > at
> > >    
> org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngi
> ne.java:538)
> > > at
> > >    
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158
> )
> > > ... 7 more
> > > [12:10:00,605][WARN][cluster.action.shard] [Magilla]
> sending
> > > failed shard for [twitter][1],
> > > node[10dab323-019b-4036-854f-89bb068dcc8d], [P],
> s[INITIALIZING], reason
> > > [Failed to start shard, message
> > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> recover
> > > translog]; nested:
> EngineCreationFailureException[[twitter][1] Failed to
> > > open reader on writer]; nested:
> > >    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/tw
> itter/1/index/_d8g.cfs
> > > (No such file or directory)]; ]]
> >
> > > Shay Banon wrote:
> > > > Also, use the latest again, pushed some more fixes.
> >
> > > > On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon
> > > > <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>> wrote:
> >
> > > > Do you see any exceptions in the logs (failing to
> start the
> > > shard)?
> >
> > > > On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> > > > <kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com> <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>> wrote:
> >
> > > > Now its looping: progress is going to 100, then
> starting
> > > over.
> >
> > > > I set up a 1/second loop using:
> > > > while /bin/true; do date; curl -XGET
> > > > 'http://192.168.1.5:9200/twitter/_status?pretty=true';
> > > sleep 1; done
> > > > then copied it to gist
> at:http://gist.github.com/540711
> >
> > > > It should have recovered by now, I would think.
> >
> > > > ...Ken
> >
> > > > Shay Banon wrote:
> > > > > great, ping me if it does not end, I am here to
> help (we can
> > > > make it
> > > > > more interactive on IRC).
> >
> > > > > p.s. Can you keep the original json format when you
> > > gist? Much
> > > > easier to
> > > > > know whats going on. You can add pretty=true as a
> > > parameter to
> > > > get it
> > > > > pretty printed.
> >
> > > > > -shay.banon
> >
> > > > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > > > > <kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> <mailto:kenneth.loaf...@gmail.com <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>>> wrote:
> >
> > > > > I think so... Here's the latest on gist
> > > > http://gist.github.com/540471
> >
> > > > > Thanks for the pointer on gist, I've never
> used it
> > > before.
> >
> > > > > Shay Banon wrote:
> > > > > > The top just states which shards were
> queries, a shard
> > > > that is
> > > > > still not
> > > > > > allocated will obviously not be allocated. It
> > > seems like
> > > > its still in
> > > > > > recovery process. There are two main APIs
> to really
> > > > understand what is
> > > > > > going on (except for the high level health
> api), the
> > > > cluster state
> > > > > API,
> > > > > > that shows you what the cluster wide state
> is (where
> > > > each shard is
> > > > > > supposed to be, what its state is), and
> the status api
> > > > which gives you
> > > > > > detailed information of the status of each
> shard
> > > > allocated on each
> > > > > node.
> >
> > > > > > Is the recovery progressing?
> >
> > > > > > p.s. Can you use gist instead of pastebin?
> >
> > > > > > -shay.banon
> >
> > > > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth
> Loafman
> > > > > > <kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>>
> > > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>
> > > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>
> > > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>
> > > <mailto:kenneth.loaf...@gmail.com
> <mailto:kenneth.loaf...@gmail.com>>>>>> wrote:
> >
> > > > > > I restarted and now 35 of 36 are
> successful,
> > > but if
> > > > you look
> > > > > at the
> > > > > > status, it's showing multiple shards
> in recovery.
> > > > I'm confused.
> >
> > > > > > See cluster status
> inhttp://pastebin.com/9qWLf3mk <http://pastebin.com/9qWLf3mk>
> >
> > > > > > Kenneth Loafman wrote:
> > > > > > > Will do so in just a bit...
> >
> > > > > > > Shay Banon wrote:
> > > > > > >> ... can you test?
> >
> > > > > > >> On Fri, Aug 20, 2010 at 4:02 PM,
> Shay Banon
> > > > > > >> <shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>
> > > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>
> > > > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>
> > > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>
> > > <mailto:shay.ba...@elasticsearch.com
> <mailto:shay.ba...@elasticsearch.com>>>>
> > > > > > <mailto:shay.ba...@elasticsearch.com.
> <mailto:shay.ba...@elasticsearch.com.>..
> >
> > read more »
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:20am UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/25 "2017-07-06T04:20:29Z")

</div>

Closed on July 6, 2017, 4:20am UTC



[Previous page](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239.md?page=1)
