# How to restart/recover a shard?

**URL:** <https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239>\
**Category:** Elasticsearch\
**Created:** [August 19, 2010, 11:28am UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239 "2010-08-19T11:28:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 19, 2010, 11:28am UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/1 "2010-08-19T11:28:33Z")

</div>

Hi,

The second shard on one of my indexes has failed due to:  
[05:59:47,332][WARN][index.gateway] [Mangog] [twitter][1]  
failed to snapshot on close  
...followed by a long traceback.  
...followed by:  
[05:59:49,336][WARN][cluster.action.shard] [Mangog] received shard  
failed for [twitter][1], node[86d601df-e124-45ed-a5f2-57d762042d87],  
[P], s[INITIALIZING], reason [Failed to start shard, message  
[IndexShardGatewayRecoveryException[[twitter][1] Failed to recovery  
translog]; nested: EngineCreationFailureException[[twitter][1] Failed to  
open reader on writer]; nested:  
FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs  
(No such file or directory)]; ]]

Is the recovery process automatic, or do I have to do something  
special? It appears to be just this one shard.

I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my config is  
below.

...Thanks,  
...Ken

cloud:  
aws:  
access\_key: \*\*\*\*\*  
secret\_key: \*\*\*\*\*

gateway:  
type: s3  
s3:  
bucket: \*\*\*\*\*

path :  
work : /mnt/search-data-dev  
logs : /mnt/search-data-dev/node1/logs

index :  
number\_of\_shards : 2  
number\_of\_replicas : 1

network :  
host : 192.168.1.5

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 19, 2010, 11:58am UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/2 "2010-08-19T11:58:31Z")

</div>

The shard will allocated to another node and recovered there. Do you see it  
happen continuously?

-shay.banon

On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Hi,
> 
> The second shard on one of my indexes has failed due to:  
> [05:59:47,332][WARN][index.gateway] [Mangog] [twitter][1]  
> failed to snapshot on close  
> ...followed by a long traceback.  
> ...followed by:  
> [05:59:49,336][WARN][cluster.action.shard] [Mangog] received shard  
> failed for [twitter][1], node[86d601df-e124-45ed-a5f2-57d762042d87],  
> [P], s[INITIALIZING], reason [Failed to start shard, message  
> [IndexShardGatewayRecoveryException[[twitter][1] Failed to recovery  
> translog]; nested: EngineCreationFailureException[[twitter][1] Failed to  
> open reader on writer]; nested:
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs  
> (No such file or directory)]; ]]
> 
> Is the recovery process automatic, or do I have to do something  
> special? It appears to be just this one shard.
> 
> I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my config is  
> below.
> 
> ...Thanks,  
> ...Ken
> 
> cloud:  
> aws:  
> access\_key: \*\*\*\*\*  
> secret\_key: \*\*\*\*\*
> 
> gateway:  
> type: s3  
> s3:  
> bucket: \*\*\*\*\*
> 
> path :  
> work : /mnt/search-data-dev  
> logs : /mnt/search-data-dev/node1/logs
> 
> index :  
> number\_of\_shards : 2  
> number\_of\_replicas : 1
> 
> network :  
> host : 192.168.1.5

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 19, 2010, 12:48pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/3 "2010-08-19T12:48:05Z")

</div>

No this is the first time. The shutdown took a while with several  
'Waiting for not to shutdown..." style message. It came up bad after that.

So, if I have two nodes now, and one needs to be recovered, I'll need 3  
nodes to get the recovery done?

...Ken

Shay Banon wrote:

> The shard will allocated to another node and recovered there. Do you see  
> it happen continuously?
> 
> -shay.banon
> 
> On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> Hi,
> 
> The second shard on one of my indexes has failed due to:
> [05:59:47,332][WARN][index.gateway] [Mangog] [twitter][1]
> failed to snapshot on close
> ...followed by a long traceback.
> ...followed by:
> [05:59:49,336][WARN][cluster.action.shard] [Mangog] received shard
> failed for [twitter][1], node[86d601df-e124-45ed-a5f2-57d762042d87],
> [P], s[INITIALIZING], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[twitter][1] Failed to recovery
> translog]; nested: EngineCreationFailureException[[twitter][1] Failed to
> open reader on writer]; nested:
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> (No such file or directory)]; ]]
> 
> Is the recovery process automatic, or do I have to do something
> special? It appears to be just this one shard.
> 
> I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my config is
> below.
> 
> ...Thanks,
> ...Ken
> 
> cloud:
> aws:
> access_key: *****
> secret_key: *****
> 
> gateway:
> type: s3
> s3:
> bucket: *****
> 
> path :
> work : /mnt/search-data-dev
> logs : /mnt/search-data-dev/node1/logs
> 
> index :
> number_of_shards : 2
> number_of_replicas : 1
> 
> network :
> host : 192.168.1.5
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 19, 2010, 2:24pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/4 "2010-08-19T14:24:30Z")

</div>

It should be allocated on the other node, you shouldn't need to start  
another node. When you issue a cluster health (simple curl can do), what is  
the status? The cluster state API gives you more information if you are  
after (each shard and its state).

On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> No this is the first time. The shutdown took a while with several  
> 'Waiting for not to shutdown..." style message. It came up bad after that.
> 
> So, if I have two nodes now, and one needs to be recovered, I'll need 3  
> nodes to get the recovery done?
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > The shard will allocated to another node and recovered there. Do you see  
> > it happen continuously?
> > 
> > -shay.banon
> > 
> > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > Hi,
> > 
> > The second shard on one of my indexes has failed due to:
> > [05:59:47,332][WARN][index.gateway] [Mangog]
> > 
> > ```
> 
> [twitter][1]
> 
> > ```
> > failed to snapshot on close
> > ...followed by a long traceback.
> > ...followed by:
> > [05:59:49,336][WARN][cluster.action.shard] [Mangog] received
> > 
> > ```
> 
> shard
> 
> > ```
> > failed for [twitter][1], node[86d601df-e124-45ed-a5f2-57d762042d87],
> > [P], s[INITIALIZING], reason [Failed to start shard, message
> > [IndexShardGatewayRecoveryException[[twitter][1] Failed to recovery
> > translog]; nested: EngineCreationFailureException[[twitter][1] Failed
> > 
> > ```
> 
> to
> 
> > ```
> > open reader on writer]; nested:
> > 
> > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > ```
> > (No such file or directory)]; ]]
> > 
> > Is the recovery process automatic, or do I have to do something
> > special? It appears to be just this one shard.
> > 
> > I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my config
> > 
> > ```
> 
> is
> 
> > ```
> > below.
> > 
> > ...Thanks,
> > ...Ken
> > 
> > cloud:
> > aws:
> > access_key: *****
> > secret_key: *****
> > 
> > gateway:
> > type: s3
> > s3:
> > bucket: *****
> > 
> > path :
> > work : /mnt/search-data-dev
> > logs : /mnt/search-data-dev/node1/logs
> > 
> > index :
> > number_of_shards : 2
> > number_of_replicas : 1
> > 
> > network :
> > host : 192.168.1.5
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 19, 2010, 9:33pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/5 "2010-08-19T21:33:32Z")

</div>

It seems to have started recover, but it's been 7.5 hours and appears to  
be stopped/hung...

> ```
> "1": [
> {
> "gateway_recovery": {
> "index": {
> "expected_recovered_size": "0b", 
> "expected_recovered_size_in_bytes": 0, 
> "recovered_size": "0b", 
> "recovered_size_in_bytes": 0, 
> "reused_size": "0b", 
> "reused_size_in_bytes": 0, 
> "size": "0b", 
> "size_in_bytes": 0, 
> "throttling_time": "0s", 
> "throttling_time_in_millis": 0
> }, 
> "stage": "RETRY", 
> "start_time_in_millis": 1282226019603, 
> "throttling_time": "7.6h", 
> "throttling_time_in_millis": 27514627, 
> "time": "7.6h", 
> "time_in_millis": 27514657, 
> "translog": {
> "recovered": 0
> }
> }, 
> "index": {
> "size": "0b", 
> "size_in_bytes": 0
> }, 
> "routing": {
> "index": "twitter", 
> "node": "031642a1-968f-40fb-b7c2-5a869769d5b4", 
> "primary": true, 
> "relocating_node": null, 
> "shard": 1, 
> "state": "INITIALIZING"
> }, 
> "state": "RECOVERING"
> }
> ]
> 
> ```

Shay Banon wrote:

> It should be allocated on the other node, you shouldn't need to start  
> another node. When you issue a cluster health (simple curl can do), what  
> is the status? The cluster state API gives you more information if you  
> are after (each shard and its state).
> 
> On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> No this is the first time. The shutdown took a while with several
> 'Waiting for not to shutdown..." style message. It came up bad
> after that.
> 
> So, if I have two nodes now, and one needs to be recovered, I'll need 3
> nodes to get the recovery done?
> 
> ...Ken
> 
> Shay Banon wrote:
> > The shard will allocated to another node and recovered there. Do
> you see
> > it happen continuously?
> >
> > -shay.banon
> >
> > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > Hi,
> >
> > The second shard on one of my indexes has failed due to:
> > [05:59:47,332][WARN][index.gateway] [Mangog]
> [twitter][1]
> > failed to snapshot on close
> > ...followed by a long traceback.
> > ...followed by:
> > [05:59:49,336][WARN][cluster.action.shard] [Mangog]
> received shard
> > failed for [twitter][1],
> node[86d601df-e124-45ed-a5f2-57d762042d87],
> > [P], s[INITIALIZING], reason [Failed to start shard, message
> > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> recovery
> > translog]; nested: EngineCreationFailureException[[twitter][1]
> Failed to
> > open reader on writer]; nested:
> >    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > (No such file or directory)]; ]]
> >
> > Is the recovery process automatic, or do I have to do something
> > special? It appears to be just this one shard.
> >
> > I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my
> config is
> > below.
> >
> > ...Thanks,
> > ...Ken
> >
> > cloud:
> > aws:
> > access_key: *****
> > secret_key: *****
> >
> > gateway:
> > type: s3
> > s3:
> > bucket: *****
> >
> > path :
> > work : /mnt/search-data-dev
> > logs : /mnt/search-data-dev/node1/logs
> >
> > index :
> > number_of_shards : 2
> > number_of_replicas : 1
> >
> > network :
> > host : 192.168.1.5
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 19, 2010, 11:29pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/6 "2010-08-19T23:29:46Z")

</div>

Do you still have it running? Can you gist the cluster state and the index  
status results?

I see that you are using master, I have fixed several things in this area,  
can you pull a new version?

-shay.banon

On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)

> wrote:

> It seems to have started recover, but it's been 7.5 hours and appears to  
> be stopped/hung...
> 
> > ```
> > "1": [
> > {
> > "gateway_recovery": {
> > "index": {
> > "expected_recovered_size": "0b",
> > "expected_recovered_size_in_bytes": 0,
> > "recovered_size": "0b",
> > "recovered_size_in_bytes": 0,
> > "reused_size": "0b",
> > "reused_size_in_bytes": 0,
> > "size": "0b",
> > "size_in_bytes": 0,
> > "throttling_time": "0s",
> > "throttling_time_in_millis": 0
> > },
> > "stage": "RETRY",
> > "start_time_in_millis": 1282226019603,
> > "throttling_time": "7.6h",
> > "throttling_time_in_millis": 27514627,
> > "time": "7.6h",
> > "time_in_millis": 27514657,
> > "translog": {
> > "recovered": 0
> > }
> > },
> > "index": {
> > "size": "0b",
> > "size_in_bytes": 0
> > },
> > "routing": {
> > "index": "twitter",
> > "node":
> > 
> > ```
> 
> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> 
> > ```
> > "primary": true,
> > "relocating_node": null,
> > "shard": 1,
> > "state": "INITIALIZING"
> > },
> > "state": "RECOVERING"
> > }
> > ]
> > 
> > ```
> 
> Shay Banon wrote:
> 
> > It should be allocated on the other node, you shouldn't need to start  
> > another node. When you issue a cluster health (simple curl can do), what  
> > is the status? The cluster state API gives you more information if you  
> > are after (each shard and its state).
> > 
> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > No this is the first time. The shutdown took a while with several
> > 'Waiting for not to shutdown..." style message. It came up bad
> > after that.
> > 
> > So, if I have two nodes now, and one needs to be recovered, I'll need
> > 
> > ```
> 
> 3
> 
> > ```
> > nodes to get the recovery done?
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > > The shard will allocated to another node and recovered there. Do
> > you see
> > > it happen continuously?
> > >
> > > -shay.banon
> > >
> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >
> > > Hi,
> > >
> > > The second shard on one of my indexes has failed due to:
> > > [05:59:47,332][WARN][index.gateway] [Mangog]
> > [twitter][1]
> > > failed to snapshot on close
> > > ...followed by a long traceback.
> > > ...followed by:
> > > [05:59:49,336][WARN][cluster.action.shard] [Mangog]
> > received shard
> > > failed for [twitter][1],
> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > [P], s[INITIALIZING], reason [Failed to start shard, message
> > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > recovery
> > > translog]; nested: EngineCreationFailureException[[twitter][1]
> > Failed to
> > > open reader on writer]; nested:
> > >
> > 
> > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > ```
> > > (No such file or directory)]; ]]
> > >
> > > Is the recovery process automatic, or do I have to do something
> > > special? It appears to be just this one shard.
> > >
> > > I use the service wrapper to start/stop 0.9.1-SNAPSHOT, and my
> > config is
> > > below.
> > >
> > > ...Thanks,
> > > ...Ken
> > >
> > > cloud:
> > > aws:
> > > access_key: *****
> > > secret_key: *****
> > >
> > > gateway:
> > > type: s3
> > > s3:
> > > bucket: *****
> > >
> > > path :
> > > work : /mnt/search-data-dev
> > > logs : /mnt/search-data-dev/node1/logs
> > >
> > > index :
> > > number_of_shards : 2
> > > number_of_replicas : 1
> > >
> > > network :
> > > host : 192.168.1.5
> > >
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 12:04pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/7 "2010-08-20T12:04:41Z")

</div>

I upgraded to last nights version, restarted, and things are worse. Now  
I have 5 shards hung at recover, not all on the same node. Weird.

I've attached the info you want. I'll leave things running for now.

...Thanks,  
...Ken

Shay Banon wrote:

> Do you still have it running? Can you gist the cluster state and the  
> index status results?
> 
> I see that you are using master, I have fixed several things in this  
> area, can you pull a new version?
> 
> -shay.banon
> 
> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> It seems to have started recover, but it's been 7.5 hours and appears to
> be stopped/hung...
> 
> > "1": [
> > {
> > "gateway_recovery": {
> > "index": {
> > "expected_recovered_size": "0b",
> > "expected_recovered_size_in_bytes": 0,
> > "recovered_size": "0b",
> > "recovered_size_in_bytes": 0,
> > "reused_size": "0b",
> > "reused_size_in_bytes": 0,
> > "size": "0b",
> > "size_in_bytes": 0,
> > "throttling_time": "0s",
> > "throttling_time_in_millis": 0
> > },
> > "stage": "RETRY",
> > "start_time_in_millis": 1282226019603,
> > "throttling_time": "7.6h",
> > "throttling_time_in_millis": 27514627,
> > "time": "7.6h",
> > "time_in_millis": 27514657,
> > "translog": {
> > "recovered": 0
> > }
> > },
> > "index": {
> > "size": "0b",
> > "size_in_bytes": 0
> > },
> > "routing": {
> > "index": "twitter",
> > "node":
> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > "primary": true,
> > "relocating_node": null,
> > "shard": 1,
> > "state": "INITIALIZING"
> > },
> > "state": "RECOVERING"
> > }
> > ]
> 
> Shay Banon wrote:
> > It should be allocated on the other node, you shouldn't need to start
> > another node. When you issue a cluster health (simple curl can
> do), what
> > is the status? The cluster state API gives you more information if you
> > are after (each shard and its state).
> >
> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > No this is the first time. The shutdown took a while with several
> > 'Waiting for not to shutdown..." style message. It came up bad
> > after that.
> >
> > So, if I have two nodes now, and one needs to be recovered,
> I'll need 3
> > nodes to get the recovery done?
> >
> > ...Ken
> >
> > Shay Banon wrote:
> > > The shard will allocated to another node and recovered there. Do
> > you see
> > > it happen continuously?
> > >
> > > -shay.banon
> > >
> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >
> > > Hi,
> > >
> > > The second shard on one of my indexes has failed due to:
> > > [05:59:47,332][WARN][index.gateway] [Mangog]
> > [twitter][1]
> > > failed to snapshot on close
> > > ...followed by a long traceback.
> > > ...followed by:
> > > [05:59:49,336][WARN][cluster.action.shard] [Mangog]
> > received shard
> > > failed for [twitter][1],
> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > [P], s[INITIALIZING], reason [Failed to start shard, message
> > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > recovery
> > > translog]; nested:
> EngineCreationFailureException[[twitter][1]
> > Failed to
> > > open reader on writer]; nested:
> > >
> >    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > > (No such file or directory)]; ]]
> > >
> > > Is the recovery process automatic, or do I have to do
> something
> > > special? It appears to be just this one shard.
> > >
> > > I use the service wrapper to start/stop 0.9.1-SNAPSHOT,
> and my
> > config is
> > > below.
> > >
> > > ...Thanks,
> > > ...Ken
> > >
> > > cloud:
> > > aws:
> > > access_key: *****
> > > secret_key: *****
> > >
> > > gateway:
> > > type: s3
> > > s3:
> > > bucket: *****
> > >
> > > path :
> > > work : /mnt/search-data-dev
> > > logs : /mnt/search-data-dev/node1/logs
> > >
> > > index :
> > > number_of_shards : 2
> > > number_of_replicas : 1
> > >
> > > network :
> > > host : 192.168.1.5
> > >
> > >
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 12:31pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/8 "2010-08-20T12:31:05Z")

</div>

Attachments did not make it. See:  
[http://pastebin.com/ziALRgx5](http://pastebin.com/ziALRgx5) -- cluster state  
[http://pastebin.com/63Xm95xM](http://pastebin.com/63Xm95xM) -- index status

Sorry, they lost their formatting on Pastbin.

...Ken

Kenneth Loafman wrote:

> I upgraded to last nights version, restarted, and things are worse. Now  
> I have 5 shards hung at recover, not all on the same node. Weird.
> 
> I've attached the info you want. I'll leave things running for now.
> 
> ...Thanks,  
> ...Ken
> 
> Shay Banon wrote:
> 
> > Do you still have it running? Can you gist the cluster state and the  
> > index status results?
> > 
> > I see that you are using master, I have fixed several things in this  
> > area, can you pull a new version?
> > 
> > -shay.banon
> > 
> > On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > It seems to have started recover, but it's been 7.5 hours and appears to
> > be stopped/hung...
> > 
> > > "1": [
> > > {
> > > "gateway_recovery": {
> > > "index": {
> > > "expected_recovered_size": "0b",
> > > "expected_recovered_size_in_bytes": 0,
> > > "recovered_size": "0b",
> > > "recovered_size_in_bytes": 0,
> > > "reused_size": "0b",
> > > "reused_size_in_bytes": 0,
> > > "size": "0b",
> > > "size_in_bytes": 0,
> > > "throttling_time": "0s",
> > > "throttling_time_in_millis": 0
> > > },
> > > "stage": "RETRY",
> > > "start_time_in_millis": 1282226019603,
> > > "throttling_time": "7.6h",
> > > "throttling_time_in_millis": 27514627,
> > > "time": "7.6h",
> > > "time_in_millis": 27514657,
> > > "translog": {
> > > "recovered": 0
> > > }
> > > },
> > > "index": {
> > > "size": "0b",
> > > "size_in_bytes": 0
> > > },
> > > "routing": {
> > > "index": "twitter",
> > > "node":
> > "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > "primary": true,
> > > "relocating_node": null,
> > > "shard": 1,
> > > "state": "INITIALIZING"
> > > },
> > > "state": "RECOVERING"
> > > }
> > > ]
> > 
> > Shay Banon wrote:
> > > It should be allocated on the other node, you shouldn't need to start
> > > another node. When you issue a cluster health (simple curl can
> > do), what
> > > is the status? The cluster state API gives you more information if you
> > > are after (each shard and its state).
> > >
> > > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >
> > > No this is the first time. The shutdown took a while with several
> > > 'Waiting for not to shutdown..." style message. It came up bad
> > > after that.
> > >
> > > So, if I have two nodes now, and one needs to be recovered,
> > I'll need 3
> > > nodes to get the recovery done?
> > >
> > > ...Ken
> > >
> > > Shay Banon wrote:
> > > > The shard will allocated to another node and recovered there. Do
> > > you see
> > > > it happen continuously?
> > > >
> > > > -shay.banon
> > > >
> > > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > >
> > > > Hi,
> > > >
> > > > The second shard on one of my indexes has failed due to:
> > > > [05:59:47,332][WARN][index.gateway] [Mangog]
> > > [twitter][1]
> > > > failed to snapshot on close
> > > > ...followed by a long traceback.
> > > > ...followed by:
> > > > [05:59:49,336][WARN][cluster.action.shard] [Mangog]
> > > received shard
> > > > failed for [twitter][1],
> > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > [P], s[INITIALIZING], reason [Failed to start shard, message
> > > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > > recovery
> > > > translog]; nested:
> > EngineCreationFailureException[[twitter][1]
> > > Failed to
> > > > open reader on writer]; nested:
> > > >
> > >    
> > FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > > > (No such file or directory)]; ]]
> > > >
> > > > Is the recovery process automatic, or do I have to do
> > something
> > > > special? It appears to be just this one shard.
> > > >
> > > > I use the service wrapper to start/stop 0.9.1-SNAPSHOT,
> > and my
> > > config is
> > > > below.
> > > >
> > > > ...Thanks,
> > > > ...Ken
> > > >
> > > > cloud:
> > > > aws:
> > > > access_key: *****
> > > > secret_key: *****
> > > >
> > > > gateway:
> > > > type: s3
> > > > s3:
> > > > bucket: *****
> > > >
> > > > path :
> > > > work : /mnt/search-data-dev
> > > > logs : /mnt/search-data-dev/node1/logs
> > > >
> > > > index :
> > > > number_of_shards : 2
> > > > number_of_replicas : 1
> > > >
> > > > network :
> > > > host : 192.168.1.5
> > > >
> > > >
> > >
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 1:02pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/9 "2010-08-20T13:02:58Z")

</div>

Just pushed a fix for this.

On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Attachments did not make it. See:  
> [http://pastebin.com/ziALRgx5](http://pastebin.com/ziALRgx5) -- cluster state  
> [http://pastebin.com/63Xm95xM](http://pastebin.com/63Xm95xM) -- index status
> 
> Sorry, they lost their formatting on Pastbin.
> 
> ...Ken
> 
> Kenneth Loafman wrote:
> 
> > I upgraded to last nights version, restarted, and things are worse. Now  
> > I have 5 shards hung at recover, not all on the same node. Weird.
> > 
> > I've attached the info you want. I'll leave things running for now.
> > 
> > ...Thanks,  
> > ...Ken
> > 
> > Shay Banon wrote:
> > 
> > > Do you still have it running? Can you gist the cluster state and the  
> > > index status results?
> > > 
> > > I see that you are using master, I have fixed several things in this  
> > > area, can you pull a new version?
> > > 
> > > -shay.banon
> > > 
> > > On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman  
> > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > 
> > > ```
> > > It seems to have started recover, but it's been 7.5 hours and
> > > 
> > > ```
> 
> appears to
> 
> > > ```
> > > be stopped/hung...
> > > 
> > > > "1": [
> > > > {
> > > > "gateway_recovery": {
> > > > "index": {
> > > > "expected_recovered_size": "0b",
> > > >
> > > 
> > > ```
> 
> "expected\_recovered\_size\_in\_bytes": 0,
> 
> > > ```
> > > > "recovered_size": "0b",
> > > > "recovered_size_in_bytes": 0,
> > > > "reused_size": "0b",
> > > > "reused_size_in_bytes": 0,
> > > > "size": "0b",
> > > > "size_in_bytes": 0,
> > > > "throttling_time": "0s",
> > > > "throttling_time_in_millis": 0
> > > > },
> > > > "stage": "RETRY",
> > > > "start_time_in_millis": 1282226019603,
> > > > "throttling_time": "7.6h",
> > > > "throttling_time_in_millis": 27514627,
> > > > "time": "7.6h",
> > > > "time_in_millis": 27514657,
> > > > "translog": {
> > > > "recovered": 0
> > > > }
> > > > },
> > > > "index": {
> > > > "size": "0b",
> > > > "size_in_bytes": 0
> > > > },
> > > > "routing": {
> > > > "index": "twitter",
> > > > "node":
> > > "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > > "primary": true,
> > > > "relocating_node": null,
> > > > "shard": 1,
> > > > "state": "INITIALIZING"
> > > > },
> > > > "state": "RECOVERING"
> > > > }
> > > > ]
> > > 
> > > Shay Banon wrote:
> > > > It should be allocated on the other node, you shouldn't need to
> > > 
> > > ```
> 
> start
> 
> > > ```
> > > > another node. When you issue a cluster health (simple curl can
> > > do), what
> > > > is the status? The cluster state API gives you more information if
> > > 
> > > ```
> 
> you
> 
> > > ```
> > > > are after (each shard and its state).
> > > >
> > > > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > > >
> > > > No this is the first time. The shutdown took a while with
> > > 
> > > ```
> 
> several
> 
> > > ```
> > > > 'Waiting for not to shutdown..." style message. It came up
> > > 
> > > ```
> 
> bad
> 
> > > ```
> > > > after that.
> > > >
> > > > So, if I have two nodes now, and one needs to be recovered,
> > > I'll need 3
> > > > nodes to get the recovery done?
> > > >
> > > > ...Ken
> > > >
> > > > Shay Banon wrote:
> > > > > The shard will allocated to another node and recovered
> > > 
> > > ```
> 
> there. Do
> 
> > > ```
> > > > you see
> > > > > it happen continuously?
> > > > >
> > > > > -shay.banon
> > > > >
> > > > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > > > <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com> <mailto:
> > > 
> > > ```
> 
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)
> 
> > > ```
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > The second shard on one of my indexes has failed due to:
> > > > > [05:59:47,332][WARN][index.gateway]
> > > 
> > > ```
> 
> [Mangog]
> 
> > > ```
> > > > [twitter][1]
> > > > > failed to snapshot on close
> > > > > ...followed by a long traceback.
> > > > > ...followed by:
> > > > > [05:59:49,336][WARN][cluster.action.shard]
> > > 
> > > ```
> 
> [Mangog]
> 
> > > ```
> > > > received shard
> > > > > failed for [twitter][1],
> > > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > > [P], s[INITIALIZING], reason [Failed to start shard,
> > > 
> > > ```
> 
> message
> 
> > > ```
> > > > > [IndexShardGatewayRecoveryException[[twitter][1] Failed
> > > 
> > > ```
> 
> to
> 
> > > ```
> > > > recovery
> > > > > translog]; nested:
> > > EngineCreationFailureException[[twitter][1]
> > > > Failed to
> > > > > open reader on writer]; nested:
> > > > >
> > > >
> > > 
> > > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > > ```
> > > > > (No such file or directory)]; ]]
> > > > >
> > > > > Is the recovery process automatic, or do I have to do
> > > something
> > > > > special? It appears to be just this one shard.
> > > > >
> > > > > I use the service wrapper to start/stop 0.9.1-SNAPSHOT,
> > > and my
> > > > config is
> > > > > below.
> > > > >
> > > > > ...Thanks,
> > > > > ...Ken
> > > > >
> > > > > cloud:
> > > > > aws:
> > > > > access_key: *****
> > > > > secret_key: *****
> > > > >
> > > > > gateway:
> > > > > type: s3
> > > > > s3:
> > > > > bucket: *****
> > > > >
> > > > > path :
> > > > > work : /mnt/search-data-dev
> > > > > logs : /mnt/search-data-dev/node1/logs
> > > > >
> > > > > index :
> > > > > number_of_shards : 2
> > > > > number_of_replicas : 1
> > > > >
> > > > > network :
> > > > > host : 192.168.1.5
> > > > >
> > > > >
> > > >
> > > >
> > > 
> > > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 1:03pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/10 "2010-08-20T13:03:05Z")

</div>

... can you test?

On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Just pushed a fix for this.
> 
> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman \<  
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> > Attachments did not make it. See:  
> > [http://pastebin.com/ziALRgx5](http://pastebin.com/ziALRgx5) -- cluster state  
> > [http://pastebin.com/63Xm95xM](http://pastebin.com/63Xm95xM) -- index status
> > 
> > Sorry, they lost their formatting on Pastbin.
> > 
> > ...Ken
> > 
> > Kenneth Loafman wrote:
> > 
> > > I upgraded to last nights version, restarted, and things are worse. Now  
> > > I have 5 shards hung at recover, not all on the same node. Weird.
> > > 
> > > I've attached the info you want. I'll leave things running for now.
> > > 
> > > ...Thanks,  
> > > ...Ken
> > > 
> > > Shay Banon wrote:
> > > 
> > > > Do you still have it running? Can you gist the cluster state and the  
> > > > index status results?
> > > > 
> > > > I see that you are using master, I have fixed several things in this  
> > > > area, can you pull a new version?
> > > > 
> > > > -shay.banon
> > > > 
> > > > On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman  
> > > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > > 
> > > > ```
> > > > It seems to have started recover, but it's been 7.5 hours and
> > > > 
> > > > ```
> > 
> > appears to
> > 
> > > > ```
> > > > be stopped/hung...
> > > > 
> > > > > "1": [
> > > > > {
> > > > > "gateway_recovery": {
> > > > > "index": {
> > > > > "expected_recovered_size": "0b",
> > > > >
> > > > 
> > > > ```
> > 
> > "expected\_recovered\_size\_in\_bytes": 0,
> > 
> > > > ```
> > > > > "recovered_size": "0b",
> > > > > "recovered_size_in_bytes": 0,
> > > > > "reused_size": "0b",
> > > > > "reused_size_in_bytes": 0,
> > > > > "size": "0b",
> > > > > "size_in_bytes": 0,
> > > > > "throttling_time": "0s",
> > > > > "throttling_time_in_millis": 0
> > > > > },
> > > > > "stage": "RETRY",
> > > > > "start_time_in_millis":
> > > > 
> > > > ```
> > 
> > 1282226019603,
> > 
> > > > ```
> > > > > "throttling_time": "7.6h",
> > > > > "throttling_time_in_millis":
> > > > 
> > > > ```
> > 
> > 27514627,
> > 
> > > > ```
> > > > > "time": "7.6h",
> > > > > "time_in_millis": 27514657,
> > > > > "translog": {
> > > > > "recovered": 0
> > > > > }
> > > > > },
> > > > > "index": {
> > > > > "size": "0b",
> > > > > "size_in_bytes": 0
> > > > > },
> > > > > "routing": {
> > > > > "index": "twitter",
> > > > > "node":
> > > > "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > > > "primary": true,
> > > > > "relocating_node": null,
> > > > > "shard": 1,
> > > > > "state": "INITIALIZING"
> > > > > },
> > > > > "state": "RECOVERING"
> > > > > }
> > > > > ]
> > > > 
> > > > Shay Banon wrote:
> > > > > It should be allocated on the other node, you shouldn't need to
> > > > 
> > > > ```
> > 
> > start
> > 
> > > > ```
> > > > > another node. When you issue a cluster health (simple curl can
> > > > do), what
> > > > > is the status? The cluster state API gives you more information
> > > > 
> > > > ```
> > 
> > if you
> > 
> > > > ```
> > > > > are after (each shard and its state).
> > > > >
> > > > > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > > > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > > > >
> > > > > No this is the first time. The shutdown took a while with
> > > > 
> > > > ```
> > 
> > several
> > 
> > > > ```
> > > > > 'Waiting for not to shutdown..." style message. It came up
> > > > 
> > > > ```
> > 
> > bad
> > 
> > > > ```
> > > > > after that.
> > > > >
> > > > > So, if I have two nodes now, and one needs to be recovered,
> > > > I'll need 3
> > > > > nodes to get the recovery done?
> > > > >
> > > > > ...Ken
> > > > >
> > > > > Shay Banon wrote:
> > > > > > The shard will allocated to another node and recovered
> > > > 
> > > > ```
> > 
> > there. Do
> > 
> > > > ```
> > > > > you see
> > > > > > it happen continuously?
> > > > > >
> > > > > > -shay.banon
> > > > > >
> > > > > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > > > > <kenneth.loafman@gmail.com
> > > > <mailto:kenneth.loafman@gmail.com> <mailto:
> > > > 
> > > > ```
> > 
> > [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)
> > 
> > > > ```
> > > > <mailto:kenneth.loafman@gmail.com>>
> > > > > <mailto:kenneth.loafman@gmail.com
> > > > <mailto:kenneth.loafman@gmail.com>
> > > > > <mailto:kenneth.loafman@gmail.com
> > > > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > The second shard on one of my indexes has failed due
> > > > 
> > > > ```
> > 
> > to:
> > 
> > > > ```
> > > > > > [05:59:47,332][WARN][index.gateway]
> > > > 
> > > > ```
> > 
> > [Mangog]
> > 
> > > > ```
> > > > > [twitter][1]
> > > > > > failed to snapshot on close
> > > > > > ...followed by a long traceback.
> > > > > > ...followed by:
> > > > > > [05:59:49,336][WARN][cluster.action.shard]
> > > > 
> > > > ```
> > 
> > [Mangog]
> > 
> > > > ```
> > > > > received shard
> > > > > > failed for [twitter][1],
> > > > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > > > [P], s[INITIALIZING], reason [Failed to start shard,
> > > > 
> > > > ```
> > 
> > message
> > 
> > > > ```
> > > > > > [IndexShardGatewayRecoveryException[[twitter][1] Failed
> > > > 
> > > > ```
> > 
> > to
> > 
> > > > ```
> > > > > recovery
> > > > > > translog]; nested:
> > > > EngineCreationFailureException[[twitter][1]
> > > > > Failed to
> > > > > > open reader on writer]; nested:
> > > > > >
> > > > >
> > > > 
> > > > ```
> > 
> > FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> > 
> > > > ```
> > > > > > (No such file or directory)]; ]]
> > > > > >
> > > > > > Is the recovery process automatic, or do I have to do
> > > > something
> > > > > > special? It appears to be just this one shard.
> > > > > >
> > > > > > I use the service wrapper to start/stop 0.9.1-SNAPSHOT,
> > > > and my
> > > > > config is
> > > > > > below.
> > > > > >
> > > > > > ...Thanks,
> > > > > > ...Ken
> > > > > >
> > > > > > cloud:
> > > > > > aws:
> > > > > > access_key: *****
> > > > > > secret_key: *****
> > > > > >
> > > > > > gateway:
> > > > > > type: s3
> > > > > > s3:
> > > > > > bucket: *****
> > > > > >
> > > > > > path :
> > > > > > work : /mnt/search-data-dev
> > > > > > logs : /mnt/search-data-dev/node1/logs
> > > > > >
> > > > > > index :
> > > > > > number_of_shards : 2
> > > > > > number_of_replicas : 1
> > > > > >
> > > > > > network :
> > > > > > host : 192.168.1.5
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > 
> > > > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 1:29pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/11 "2010-08-20T13:29:56Z")

</div>

Will do so in just a bit...

Shay Banon wrote:

> ... can you test?
> 
> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon  
> \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) [mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> 
> ```
> Just pushed a fix for this.
> 
> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>> wrote:
> 
> Attachments did not make it. See:
> http://pastebin.com/ziALRgx5 -- cluster state
> http://pastebin.com/63Xm95xM -- index status
> 
> Sorry, they lost their formatting on Pastbin.
> 
> ...Ken
> 
> Kenneth Loafman wrote:
> > I upgraded to last nights version, restarted, and things are
> worse. Now
> > I have 5 shards hung at recover, not all on the same node. Weird.
> >
> > I've attached the info you want. I'll leave things running
> for now.
> >
> > ...Thanks,
> > ...Ken
> >
> > Shay Banon wrote:
> >> Do you still have it running? Can you gist the cluster state
> and the
> >> index status results?
> >>
> >> I see that you are using master, I have fixed several things
> in this
> >> area, can you pull a new version?
> >>
> >> -shay.banon
> >>
> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> >> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >>
> >> It seems to have started recover, but it's been 7.5 hours
> and appears to
> >> be stopped/hung...
> >>
> >> > "1": [
> >> > {
> >> > "gateway_recovery": {
> >> > "index": {
> >> >                                
> "expected_recovered_size": "0b",
> >> >                                
> "expected_recovered_size_in_bytes": 0,
> >> > "recovered_size": "0b",
> >> >                                
> "recovered_size_in_bytes": 0,
> >> > "reused_size": "0b",
> >> > "reused_size_in_bytes": 0,
> >> > "size": "0b",
> >> > "size_in_bytes": 0,
> >> > "throttling_time": "0s",
> >> >                                
> "throttling_time_in_millis": 0
> >> > },
> >> > "stage": "RETRY",
> >> > "start_time_in_millis":
> 1282226019603,
> >> > "throttling_time": "7.6h",
> >> >                            
> "throttling_time_in_millis": 27514627,
> >> > "time": "7.6h",
> >> > "time_in_millis": 27514657,
> >> > "translog": {
> >> > "recovered": 0
> >> > }
> >> > },
> >> > "index": {
> >> > "size": "0b",
> >> > "size_in_bytes": 0
> >> > },
> >> > "routing": {
> >> > "index": "twitter",
> >> > "node":
> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> >> > "primary": true,
> >> > "relocating_node": null,
> >> > "shard": 1,
> >> > "state": "INITIALIZING"
> >> > },
> >> > "state": "RECOVERING"
> >> > }
> >> > ]
> >>
> >>
> >> Shay Banon wrote:
> >> > It should be allocated on the other node, you shouldn't
> need to start
> >> > another node. When you issue a cluster health (simple
> curl can
> >> do), what
> >> > is the status? The cluster state API gives you more
> information if you
> >> > are after (each shard and its state).
> >> >
> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> >> > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> >> >
> >> > No this is the first time. The shutdown took a
> while with several
> >> > 'Waiting for not to shutdown..." style message. It
> came up bad
> >> > after that.
> >> >
> >> > So, if I have two nodes now, and one needs to be
> recovered,
> >> I'll need 3
> >> > nodes to get the recovery done?
> >> >
> >> > ...Ken
> >> >
> >> > Shay Banon wrote:
> >> > > The shard will allocated to another node and
> recovered there. Do
> >> > you see
> >> > > it happen continuously?
> >> > >
> >> > > -shay.banon
> >> > >
> >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> >> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> >> > >
> >> > > Hi,
> >> > >
> >> > > The second shard on one of my indexes has
> failed due to:
> >> > > [05:59:47,332][WARN][index.gateway          
> ] [Mangog]
> >> > [twitter][1]
> >> > > failed to snapshot on close
> >> > > ...followed by a long traceback.
> >> > > ...followed by:
> >> > > [05:59:49,336][WARN][cluster.action.shard  
> ] [Mangog]
> >> > received shard
> >> > > failed for [twitter][1],
> >> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> >> > > [P], s[INITIALIZING], reason [Failed to start
> shard, message
> >> > >    
> [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> >> > recovery
> >> > > translog]; nested:
> >> EngineCreationFailureException[[twitter][1]
> >> > Failed to
> >> > > open reader on writer]; nested:
> >> > >
> >> >
> >>    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> >> > > (No such file or directory)]; ]]
> >> > >
> >> > > Is the recovery process automatic, or do I
> have to do
> >> something
> >> > > special? It appears to be just this one shard.
> >> > >
> >> > > I use the service wrapper to start/stop
> 0.9.1-SNAPSHOT,
> >> and my
> >> > config is
> >> > > below.
> >> > >
> >> > > ...Thanks,
> >> > > ...Ken
> >> > >
> >> > > cloud:
> >> > > aws:
> >> > > access_key: *****
> >> > > secret_key: *****
> >> > >
> >> > > gateway:
> >> > > type: s3
> >> > > s3:
> >> > > bucket: *****
> >> > >
> >> > > path :
> >> > > work : /mnt/search-data-dev
> >> > > logs : /mnt/search-data-dev/node1/logs
> >> > >
> >> > > index :
> >> > > number_of_shards : 2
> >> > > number_of_replicas : 1
> >> > >
> >> > > network :
> >> > > host : 192.168.1.5
> >> > >
> >> > >
> >> >
> >> >
> >>
> >>
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 2:13pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/12 "2010-08-20T14:13:50Z")

</div>

I restarted and now 35 of 36 are successful, but if you look at the  
status, it's showing multiple shards in recovery. I'm confused.

See cluster status in [http://pastebin.com/9qWLf3mk](http://pastebin.com/9qWLf3mk)

Kenneth Loafman wrote:

> Will do so in just a bit...
> 
> Shay Banon wrote:
> 
> > ... can you test?
> > 
> > On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon  
> > \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) [mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> > 
> > ```
> > Just pushed a fix for this.
> > 
> > On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>> wrote:
> > 
> > Attachments did not make it. See:
> > http://pastebin.com/ziALRgx5 -- cluster state
> > http://pastebin.com/63Xm95xM -- index status
> > 
> > Sorry, they lost their formatting on Pastbin.
> > 
> > ...Ken
> > 
> > Kenneth Loafman wrote:
> > > I upgraded to last nights version, restarted, and things are
> > worse. Now
> > > I have 5 shards hung at recover, not all on the same node. Weird.
> > >
> > > I've attached the info you want. I'll leave things running
> > for now.
> > >
> > > ...Thanks,
> > > ...Ken
> > >
> > > Shay Banon wrote:
> > >> Do you still have it running? Can you gist the cluster state
> > and the
> > >> index status results?
> > >>
> > >> I see that you are using master, I have fixed several things
> > in this
> > >> area, can you pull a new version?
> > >>
> > >> -shay.banon
> > >>
> > >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> > >> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >>
> > >> It seems to have started recover, but it's been 7.5 hours
> > and appears to
> > >> be stopped/hung...
> > >>
> > >> > "1": [
> > >> > {
> > >> > "gateway_recovery": {
> > >> > "index": {
> > >> >                                
> > "expected_recovered_size": "0b",
> > >> >                                
> > "expected_recovered_size_in_bytes": 0,
> > >> > "recovered_size": "0b",
> > >> >                                
> > "recovered_size_in_bytes": 0,
> > >> > "reused_size": "0b",
> > >> > "reused_size_in_bytes": 0,
> > >> > "size": "0b",
> > >> > "size_in_bytes": 0,
> > >> > "throttling_time": "0s",
> > >> >                                
> > "throttling_time_in_millis": 0
> > >> > },
> > >> > "stage": "RETRY",
> > >> > "start_time_in_millis":
> > 1282226019603,
> > >> > "throttling_time": "7.6h",
> > >> >                            
> > "throttling_time_in_millis": 27514627,
> > >> > "time": "7.6h",
> > >> > "time_in_millis": 27514657,
> > >> > "translog": {
> > >> > "recovered": 0
> > >> > }
> > >> > },
> > >> > "index": {
> > >> > "size": "0b",
> > >> > "size_in_bytes": 0
> > >> > },
> > >> > "routing": {
> > >> > "index": "twitter",
> > >> > "node":
> > >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > >> > "primary": true,
> > >> > "relocating_node": null,
> > >> > "shard": 1,
> > >> > "state": "INITIALIZING"
> > >> > },
> > >> > "state": "RECOVERING"
> > >> > }
> > >> > ]
> > >>
> > >>
> > >> Shay Banon wrote:
> > >> > It should be allocated on the other node, you shouldn't
> > need to start
> > >> > another node. When you issue a cluster health (simple
> > curl can
> > >> do), what
> > >> > is the status? The cluster state API gives you more
> > information if you
> > >> > are after (each shard and its state).
> > >> >
> > >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > >> > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >> >
> > >> > No this is the first time. The shutdown took a
> > while with several
> > >> > 'Waiting for not to shutdown..." style message. It
> > came up bad
> > >> > after that.
> > >> >
> > >> > So, if I have two nodes now, and one needs to be
> > recovered,
> > >> I'll need 3
> > >> > nodes to get the recovery done?
> > >> >
> > >> > ...Ken
> > >> >
> > >> > Shay Banon wrote:
> > >> > > The shard will allocated to another node and
> > recovered there. Do
> > >> > you see
> > >> > > it happen continuously?
> > >> > >
> > >> > > -shay.banon
> > >> > >
> > >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > >> > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > >> > >
> > >> > > Hi,
> > >> > >
> > >> > > The second shard on one of my indexes has
> > failed due to:
> > >> > > [05:59:47,332][WARN][index.gateway          
> > ] [Mangog]
> > >> > [twitter][1]
> > >> > > failed to snapshot on close
> > >> > > ...followed by a long traceback.
> > >> > > ...followed by:
> > >> > > [05:59:49,336][WARN][cluster.action.shard  
> > ] [Mangog]
> > >> > received shard
> > >> > > failed for [twitter][1],
> > >> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > >> > > [P], s[INITIALIZING], reason [Failed to start
> > shard, message
> > >> > >    
> > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > >> > recovery
> > >> > > translog]; nested:
> > >> EngineCreationFailureException[[twitter][1]
> > >> > Failed to
> > >> > > open reader on writer]; nested:
> > >> > >
> > >> >
> > >>    
> > FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > >> > > (No such file or directory)]; ]]
> > >> > >
> > >> > > Is the recovery process automatic, or do I
> > have to do
> > >> something
> > >> > > special? It appears to be just this one shard.
> > >> > >
> > >> > > I use the service wrapper to start/stop
> > 0.9.1-SNAPSHOT,
> > >> and my
> > >> > config is
> > >> > > below.
> > >> > >
> > >> > > ...Thanks,
> > >> > > ...Ken
> > >> > >
> > >> > > cloud:
> > >> > > aws:
> > >> > > access_key: *****
> > >> > > secret_key: *****
> > >> > >
> > >> > > gateway:
> > >> > > type: s3
> > >> > > s3:
> > >> > > bucket: *****
> > >> > >
> > >> > > path :
> > >> > > work : /mnt/search-data-dev
> > >> > > logs : /mnt/search-data-dev/node1/logs
> > >> > >
> > >> > > index :
> > >> > > number_of_shards : 2
> > >> > > number_of_replicas : 1
> > >> > >
> > >> > > network :
> > >> > > host : 192.168.1.5
> > >> > >
> > >> > >
> > >> >
> > >> >
> > >>
> > >>
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 2:30pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/13 "2010-08-20T14:30:34Z")

</div>

The top just states which shards were queries, a shard that is still not  
allocated will obviously not be allocated. It seems like its still in  
recovery process. There are two main APIs to really understand what is going  
on (except for the high level health api), the cluster state API, that shows  
you what the cluster wide state is (where each shard is supposed to be, what  
its state is), and the status api which gives you detailed information of  
the status of each shard allocated on each node.

Is the recovery progressing?

p.s. Can you use gist instead of pastebin?

-shay.banon

On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> I restarted and now 35 of 36 are successful, but if you look at the  
> status, it's showing multiple shards in recovery. I'm confused.
> 
> See cluster status in [http://pastebin.com/9qWLf3mk](http://pastebin.com/9qWLf3mk)
> 
> Kenneth Loafman wrote:
> 
> > Will do so in just a bit...
> > 
> > Shay Banon wrote:
> > 
> > > ... can you test?
> > > 
> > > On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon  
> > > \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) [mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\>  
> > > wrote:
> > > 
> > > ```
> > > Just pushed a fix for this.
> > > 
> > > On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>>
> > > 
> > > ```
> 
> wrote:
> 
> > > ```
> > > Attachments did not make it. See:
> > > http://pastebin.com/ziALRgx5 -- cluster state
> > > http://pastebin.com/63Xm95xM -- index status
> > > 
> > > Sorry, they lost their formatting on Pastbin.
> > > 
> > > ...Ken
> > > 
> > > Kenneth Loafman wrote:
> > > > I upgraded to last nights version, restarted, and things are
> > > worse. Now
> > > > I have 5 shards hung at recover, not all on the same node.
> > > 
> > > ```
> 
> Weird.
> 
> > > ```
> > > >
> > > > I've attached the info you want. I'll leave things running
> > > for now.
> > > >
> > > > ...Thanks,
> > > > ...Ken
> > > >
> > > > Shay Banon wrote:
> > > >> Do you still have it running? Can you gist the cluster state
> > > and the
> > > >> index status results?
> > > >>
> > > >> I see that you are using master, I have fixed several things
> > > in this
> > > >> area, can you pull a new version?
> > > >>
> > > >> -shay.banon
> > > >>
> > > >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> > > >> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com
> > > 
> > > ```
> > 
> > > ```
> > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > > >>
> > > >> It seems to have started recover, but it's been 7.5 hours
> > > and appears to
> > > >> be stopped/hung...
> > > >>
> > > >> > "1": [
> > > >> > {
> > > >> > "gateway_recovery": {
> > > >> > "index": {
> > > >> >
> > > "expected_recovered_size": "0b",
> > > >> >
> > > "expected_recovered_size_in_bytes": 0,
> > > >> > "recovered_size": "0b",
> > > >> >
> > > "recovered_size_in_bytes": 0,
> > > >> > "reused_size": "0b",
> > > >> > "reused_size_in_bytes":
> > > 
> > > ```
> 
> 0,
> 
> > > ```
> > > >> > "size": "0b",
> > > >> > "size_in_bytes": 0,
> > > >> > "throttling_time":
> > > 
> > > ```
> 
> "0s",
> 
> > > ```
> > > >> >
> > > "throttling_time_in_millis": 0
> > > >> > },
> > > >> > "stage": "RETRY",
> > > >> > "start_time_in_millis":
> > > 1282226019603,
> > > >> > "throttling_time": "7.6h",
> > > >> >
> > > "throttling_time_in_millis": 27514627,
> > > >> > "time": "7.6h",
> > > >> > "time_in_millis": 27514657,
> > > >> > "translog": {
> > > >> > "recovered": 0
> > > >> > }
> > > >> > },
> > > >> > "index": {
> > > >> > "size": "0b",
> > > >> > "size_in_bytes": 0
> > > >> > },
> > > >> > "routing": {
> > > >> > "index": "twitter",
> > > >> > "node":
> > > >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > >> > "primary": true,
> > > >> > "relocating_node": null,
> > > >> > "shard": 1,
> > > >> > "state": "INITIALIZING"
> > > >> > },
> > > >> > "state": "RECOVERING"
> > > >> > }
> > > >> > ]
> > > >>
> > > >>
> > > >> Shay Banon wrote:
> > > >> > It should be allocated on the other node, you shouldn't
> > > need to start
> > > >> > another node. When you issue a cluster health (simple
> > > curl can
> > > >> do), what
> > > >> > is the status? The cluster state API gives you more
> > > information if you
> > > >> > are after (each shard and its state).
> > > >> >
> > > >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > > >> > <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > >> >
> > > >> > No this is the first time. The shutdown took a
> > > while with several
> > > >> > 'Waiting for not to shutdown..." style message. It
> > > came up bad
> > > >> > after that.
> > > >> >
> > > >> > So, if I have two nodes now, and one needs to be
> > > recovered,
> > > >> I'll need 3
> > > >> > nodes to get the recovery done?
> > > >> >
> > > >> > ...Ken
> > > >> >
> > > >> > Shay Banon wrote:
> > > >> > > The shard will allocated to another node and
> > > recovered there. Do
> > > >> > you see
> > > >> > > it happen continuously?
> > > >> > >
> > > >> > > -shay.banon
> > > >> > >
> > > >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth Loafman
> > > >> > > <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com <mailto:
> > > 
> > > ```
> 
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\>
> 
> > > ```
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > >> > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > >> > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > >> > >
> > > >> > > Hi,
> > > >> > >
> > > >> > > The second shard on one of my indexes has
> > > failed due to:
> > > >> > > [05:59:47,332][WARN][index.gateway
> > > ] [Mangog]
> > > >> > [twitter][1]
> > > >> > > failed to snapshot on close
> > > >> > > ...followed by a long traceback.
> > > >> > > ...followed by:
> > > >> > > [05:59:49,336][WARN][cluster.action.shard
> > > ] [Mangog]
> > > >> > received shard
> > > >> > > failed for [twitter][1],
> > > >> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > >> > > [P], s[INITIALIZING], reason [Failed to start
> > > shard, message
> > > >> > >
> > > [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > > >> > recovery
> > > >> > > translog]; nested:
> > > >> EngineCreationFailureException[[twitter][1]
> > > >> > Failed to
> > > >> > > open reader on writer]; nested:
> > > >> > >
> > > >> >
> > > >>
> > > 
> > > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > > ```
> > > >> > > (No such file or directory)]; ]]
> > > >> > >
> > > >> > > Is the recovery process automatic, or do I
> > > have to do
> > > >> something
> > > >> > > special? It appears to be just this one
> > > 
> > > ```
> 
> shard.
> 
> > > ```
> > > >> > >
> > > >> > > I use the service wrapper to start/stop
> > > 0.9.1-SNAPSHOT,
> > > >> and my
> > > >> > config is
> > > >> > > below.
> > > >> > >
> > > >> > > ...Thanks,
> > > >> > > ...Ken
> > > >> > >
> > > >> > > cloud:
> > > >> > > aws:
> > > >> > > access_key: *****
> > > >> > > secret_key: *****
> > > >> > >
> > > >> > > gateway:
> > > >> > > type: s3
> > > >> > > s3:
> > > >> > > bucket: *****
> > > >> > >
> > > >> > > path :
> > > >> > > work : /mnt/search-data-dev
> > > >> > > logs : /mnt/search-data-dev/node1/logs
> > > >> > >
> > > >> > > index :
> > > >> > > number_of_shards : 2
> > > >> > > number_of_replicas : 1
> > > >> > >
> > > >> > > network :
> > > >> > > host : 192.168.1.5
> > > >> > >
> > > >> > >
> > > >> >
> > > >> >
> > > >>
> > > >>
> > > >
> > > 
> > > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 2:51pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/14 "2010-08-20T14:51:40Z")

</div>

I think so... Here's the latest on gist [index status · GitHub](http://gist.github.com/540471)

Thanks for the pointer on gist, I've never used it before.

Shay Banon wrote:

> The top just states which shards were queries, a shard that is still not  
> allocated will obviously not be allocated. It seems like its still in  
> recovery process. There are two main APIs to really understand what is  
> going on (except for the high level health api), the cluster state API,  
> that shows you what the cluster wide state is (where each shard is  
> supposed to be, what its state is), and the status api which gives you  
> detailed information of the status of each shard allocated on each node.
> 
> Is the recovery progressing?
> 
> p.s. Can you use gist instead of pastebin?
> 
> -shay.banon
> 
> On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> I restarted and now 35 of 36 are successful, but if you look at the
> status, it's showing multiple shards in recovery. I'm confused.
> 
> See cluster status in http://pastebin.com/9qWLf3mk
> 
> Kenneth Loafman wrote:
> > Will do so in just a bit...
> >
> > Shay Banon wrote:
> >> ... can you test?
> >>
> >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> >> <shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>> wrote:
> >>
> >> Just pushed a fix for this.
> >>
> >>
> >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> >> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >>
> >> Attachments did not make it. See:
> >> http://pastebin.com/ziALRgx5 -- cluster state
> >> http://pastebin.com/63Xm95xM -- index status
> >>
> >> Sorry, they lost their formatting on Pastbin.
> >>
> >> ...Ken
> >>
> >> Kenneth Loafman wrote:
> >> > I upgraded to last nights version, restarted, and
> things are
> >> worse. Now
> >> > I have 5 shards hung at recover, not all on the same
> node. Weird.
> >> >
> >> > I've attached the info you want. I'll leave things running
> >> for now.
> >> >
> >> > ...Thanks,
> >> > ...Ken
> >> >
> >> > Shay Banon wrote:
> >> >> Do you still have it running? Can you gist the cluster
> state
> >> and the
> >> >> index status results?
> >> >>
> >> >> I see that you are using master, I have fixed several
> things
> >> in this
> >> >> area, can you pull a new version?
> >> >>
> >> >> -shay.banon
> >> >>
> >> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> >> >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> >> >>
> >> >> It seems to have started recover, but it's been
> 7.5 hours
> >> and appears to
> >> >> be stopped/hung...
> >> >>
> >> >> > "1": [
> >> >> > {
> >> >> > "gateway_recovery": {
> >> >> > "index": {
> >> >> >
> >> "expected_recovered_size": "0b",
> >> >> >
> >> "expected_recovered_size_in_bytes": 0,
> >> >> >                                
> "recovered_size": "0b",
> >> >> >
> >> "recovered_size_in_bytes": 0,
> >> >> > "reused_size": "0b",
> >> >> >                                
> "reused_size_in_bytes": 0,
> >> >> > "size": "0b",
> >> >> > "size_in_bytes": 0,
> >> >> >                                
> "throttling_time": "0s",
> >> >> >
> >> "throttling_time_in_millis": 0
> >> >> > },
> >> >> > "stage": "RETRY",
> >> >> > "start_time_in_millis":
> >> 1282226019603,
> >> >> > "throttling_time":
> "7.6h",
> >> >> >
> >> "throttling_time_in_millis": 27514627,
> >> >> > "time": "7.6h",
> >> >> > "time_in_millis":
> 27514657,
> >> >> > "translog": {
> >> >> > "recovered": 0
> >> >> > }
> >> >> > },
> >> >> > "index": {
> >> >> > "size": "0b",
> >> >> > "size_in_bytes": 0
> >> >> > },
> >> >> > "routing": {
> >> >> > "index": "twitter",
> >> >> > "node":
> >> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> >> >> > "primary": true,
> >> >> > "relocating_node": null,
> >> >> > "shard": 1,
> >> >> > "state": "INITIALIZING"
> >> >> > },
> >> >> > "state": "RECOVERING"
> >> >> > }
> >> >> > ]
> >> >>
> >> >>
> >> >> Shay Banon wrote:
> >> >> > It should be allocated on the other node, you
> shouldn't
> >> need to start
> >> >> > another node. When you issue a cluster health
> (simple
> >> curl can
> >> >> do), what
> >> >> > is the status? The cluster state API gives you more
> >> information if you
> >> >> > are after (each shard and its state).
> >> >> >
> >> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> >> >> > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> >> >> >
> >> >> > No this is the first time. The shutdown took a
> >> while with several
> >> >> > 'Waiting for not to shutdown..." style
> message. It
> >> came up bad
> >> >> > after that.
> >> >> >
> >> >> > So, if I have two nodes now, and one needs to be
> >> recovered,
> >> >> I'll need 3
> >> >> > nodes to get the recovery done?
> >> >> >
> >> >> > ...Ken
> >> >> >
> >> >> > Shay Banon wrote:
> >> >> > > The shard will allocated to another node and
> >> recovered there. Do
> >> >> > you see
> >> >> > > it happen continuously?
> >> >> > >
> >> >> > > -shay.banon
> >> >> > >
> >> >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth
> Loafman
> >> >> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> >> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> >> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> >> >> > >
> >> >> > > Hi,
> >> >> > >
> >> >> > > The second shard on one of my indexes has
> >> failed due to:
> >> >> > > [05:59:47,332][WARN][index.gateway
> >> ] [Mangog]
> >> >> > [twitter][1]
> >> >> > > failed to snapshot on close
> >> >> > > ...followed by a long traceback.
> >> >> > > ...followed by:
> >> >> > > [05:59:49,336][WARN][cluster.action.shard
> >> ] [Mangog]
> >> >> > received shard
> >> >> > > failed for [twitter][1],
> >> >> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> >> >> > > [P], s[INITIALIZING], reason [Failed
> to start
> >> shard, message
> >> >> > >
> >> [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> >> >> > recovery
> >> >> > > translog]; nested:
> >> >> EngineCreationFailureException[[twitter][1]
> >> >> > Failed to
> >> >> > > open reader on writer]; nested:
> >> >> > >
> >> >> >
> >> >>
> >>        
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> >> >> > > (No such file or directory)]; ]]
> >> >> > >
> >> >> > > Is the recovery process automatic, or do I
> >> have to do
> >> >> something
> >> >> > > special? It appears to be just this
> one shard.
> >> >> > >
> >> >> > > I use the service wrapper to start/stop
> >> 0.9.1-SNAPSHOT,
> >> >> and my
> >> >> > config is
> >> >> > > below.
> >> >> > >
> >> >> > > ...Thanks,
> >> >> > > ...Ken
> >> >> > >
> >> >> > > cloud:
> >> >> > > aws:
> >> >> > > access_key: *****
> >> >> > > secret_key: *****
> >> >> > >
> >> >> > > gateway:
> >> >> > > type: s3
> >> >> > > s3:
> >> >> > > bucket: *****
> >> >> > >
> >> >> > > path :
> >> >> > > work : /mnt/search-data-dev
> >> >> > > logs : /mnt/search-data-dev/node1/logs
> >> >> > >
> >> >> > > index :
> >> >> > > number_of_shards : 2
> >> >> > > number_of_replicas : 1
> >> >> > >
> >> >> > > network :
> >> >> > > host : 192.168.1.5
> >> >> > >
> >> >> > >
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >
> >>
> >>
> >>
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 2:59pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/15 "2010-08-20T14:59:32Z")

</div>

great, ping me if it does not end, I am here to help (we can make it more  
interactive on IRC).

p.s. Can you keep the original json format when you gist? Much easier to  
know whats going on. You can add pretty=true as a parameter to get it pretty  
printed.

-shay.banon

On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> I think so... Here's the latest on gist [index status · GitHub](http://gist.github.com/540471)
> 
> Thanks for the pointer on gist, I've never used it before.
> 
> Shay Banon wrote:
> 
> > The top just states which shards were queries, a shard that is still not  
> > allocated will obviously not be allocated. It seems like its still in  
> > recovery process. There are two main APIs to really understand what is  
> > going on (except for the high level health api), the cluster state API,  
> > that shows you what the cluster wide state is (where each shard is  
> > supposed to be, what its state is), and the status api which gives you  
> > detailed information of the status of each shard allocated on each node.
> > 
> > Is the recovery progressing?
> > 
> > p.s. Can you use gist instead of pastebin?
> > 
> > -shay.banon
> > 
> > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > I restarted and now 35 of 36 are successful, but if you look at the
> > status, it's showing multiple shards in recovery. I'm confused.
> > 
> > See cluster status in http://pastebin.com/9qWLf3mk
> > 
> > Kenneth Loafman wrote:
> > > Will do so in just a bit...
> > >
> > > Shay Banon wrote:
> > >> ... can you test?
> > >>
> > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > >> <shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>> wrote:
> > >>
> > >> Just pushed a fix for this.
> > >>
> > >>
> > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > >> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >>
> > >> Attachments did not make it. See:
> > >> http://pastebin.com/ziALRgx5 -- cluster state
> > >> http://pastebin.com/63Xm95xM -- index status
> > >>
> > >> Sorry, they lost their formatting on Pastbin.
> > >>
> > >> ...Ken
> > >>
> > >> Kenneth Loafman wrote:
> > >> > I upgraded to last nights version, restarted, and
> > things are
> > >> worse. Now
> > >> > I have 5 shards hung at recover, not all on the same
> > node. Weird.
> > >> >
> > >> > I've attached the info you want. I'll leave things
> > 
> > ```
> 
> running
> 
> > ```
> > >> for now.
> > >> >
> > >> > ...Thanks,
> > >> > ...Ken
> > >> >
> > >> > Shay Banon wrote:
> > >> >> Do you still have it running? Can you gist the cluster
> > state
> > >> and the
> > >> >> index status results?
> > >> >>
> > >> >> I see that you are using master, I have fixed several
> > things
> > >> in this
> > >> >> area, can you pull a new version?
> > >> >>
> > >> >> -shay.banon
> > >> >>
> > >> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> > >> >> <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >> >>
> > >> >> It seems to have started recover, but it's been
> > 7.5 hours
> > >> and appears to
> > >> >> be stopped/hung...
> > >> >>
> > >> >> > "1": [
> > >> >> > {
> > >> >> > "gateway_recovery": {
> > >> >> > "index": {
> > >> >> >
> > >> "expected_recovered_size": "0b",
> > >> >> >
> > >> "expected_recovered_size_in_bytes": 0,
> > >> >> >
> > "recovered_size": "0b",
> > >> >> >
> > >> "recovered_size_in_bytes": 0,
> > >> >> > "reused_size":
> > 
> > ```
> 
> "0b",
> 
> > ```
> > >> >> >
> > "reused_size_in_bytes": 0,
> > >> >> > "size": "0b",
> > >> >> > "size_in_bytes":
> > 
> > ```
> 
> 0,
> 
> > ```
> > >> >> >
> > "throttling_time": "0s",
> > >> >> >
> > >> "throttling_time_in_millis": 0
> > >> >> > },
> > >> >> > "stage": "RETRY",
> > >> >> >
> > 
> > ```
> 
> "start\_time\_in\_millis":
> 
> > ```
> > >> 1282226019603,
> > >> >> > "throttling_time":
> > "7.6h",
> > >> >> >
> > >> "throttling_time_in_millis": 27514627,
> > >> >> > "time": "7.6h",
> > >> >> > "time_in_millis":
> > 27514657,
> > >> >> > "translog": {
> > >> >> > "recovered": 0
> > >> >> > }
> > >> >> > },
> > >> >> > "index": {
> > >> >> > "size": "0b",
> > >> >> > "size_in_bytes": 0
> > >> >> > },
> > >> >> > "routing": {
> > >> >> > "index": "twitter",
> > >> >> > "node":
> > >> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > >> >> > "primary": true,
> > >> >> > "relocating_node":
> > 
> > ```
> 
> null,
> 
> > ```
> > >> >> > "shard": 1,
> > >> >> > "state":
> > 
> > ```
> 
> "INITIALIZING"
> 
> > ```
> > >> >> > },
> > >> >> > "state": "RECOVERING"
> > >> >> > }
> > >> >> > ]
> > >> >>
> > >> >>
> > >> >> Shay Banon wrote:
> > >> >> > It should be allocated on the other node, you
> > shouldn't
> > >> need to start
> > >> >> > another node. When you issue a cluster health
> > (simple
> > >> curl can
> > >> >> do), what
> > >> >> > is the status? The cluster state API gives you
> > 
> > ```
> 
> more
> 
> > ```
> > >> information if you
> > >> >> > are after (each shard and its state).
> > >> >> >
> > >> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth Loafman
> > >> >> > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > >> >> >
> > >> >> > No this is the first time. The shutdown took
> > 
> > ```
> 
> a
> 
> > ```
> > >> while with several
> > >> >> > 'Waiting for not to shutdown..." style
> > message. It
> > >> came up bad
> > >> >> > after that.
> > >> >> >
> > >> >> > So, if I have two nodes now, and one needs to
> > 
> > ```
> 
> be
> 
> > ```
> > >> recovered,
> > >> >> I'll need 3
> > >> >> > nodes to get the recovery done?
> > >> >> >
> > >> >> > ...Ken
> > >> >> >
> > >> >> > Shay Banon wrote:
> > >> >> > > The shard will allocated to another node
> > 
> > ```
> 
> and
> 
> > ```
> > >> recovered there. Do
> > >> >> > you see
> > >> >> > > it happen continuously?
> > >> >> > >
> > >> >> > > -shay.banon
> > >> >> > >
> > >> >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth
> > Loafman
> > >> >> > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > >> >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > >> >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > >> >> > >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > > The second shard on one of my indexes
> > 
> > ```
> 
> has
> 
> > ```
> > >> failed due to:
> > >> >> > > [05:59:47,332][WARN][index.gateway
> > >> ] [Mangog]
> > >> >> > [twitter][1]
> > >> >> > > failed to snapshot on close
> > >> >> > > ...followed by a long traceback.
> > >> >> > > ...followed by:
> > >> >> > > [05:59:49,336][WARN
> > 
> > ```
> 
> ][cluster.action.shard
> 
> > ```
> > >> ] [Mangog]
> > >> >> > received shard
> > >> >> > > failed for [twitter][1],
> > >> >> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > >> >> > > [P], s[INITIALIZING], reason [Failed
> > to start
> > >> shard, message
> > >> >> > >
> > >> [IndexShardGatewayRecoveryException[[twitter][1] Failed to
> > >> >> > recovery
> > >> >> > > translog]; nested:
> > >> >> EngineCreationFailureException[[twitter][1]
> > >> >> > Failed to
> > >> >> > > open reader on writer]; nested:
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> > 
> > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > ```
> > >> >> > > (No such file or directory)]; ]]
> > >> >> > >
> > >> >> > > Is the recovery process automatic, or
> > 
> > ```
> 
> do I
> 
> > ```
> > >> have to do
> > >> >> something
> > >> >> > > special? It appears to be just this
> > one shard.
> > >> >> > >
> > >> >> > > I use the service wrapper to start/stop
> > >> 0.9.1-SNAPSHOT,
> > >> >> and my
> > >> >> > config is
> > >> >> > > below.
> > >> >> > >
> > >> >> > > ...Thanks,
> > >> >> > > ...Ken
> > >> >> > >
> > >> >> > > cloud:
> > >> >> > > aws:
> > >> >> > > access_key: *****
> > >> >> > > secret_key: *****
> > >> >> > >
> > >> >> > > gateway:
> > >> >> > > type: s3
> > >> >> > > s3:
> > >> >> > > bucket: *****
> > >> >> > >
> > >> >> > > path :
> > >> >> > > work : /mnt/search-data-dev
> > >> >> > > logs :
> > 
> > ```
> 
> /mnt/search-data-dev/node1/logs
> 
> > ```
> > >> >> > >
> > >> >> > > index :
> > >> >> > > number_of_shards : 2
> > >> >> > > number_of_replicas : 1
> > >> >> > >
> > >> >> > > network :
> > >> >> > > host : 192.168.1.5
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >>
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 5:02pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/16 "2010-08-20T17:02:55Z")

</div>

Now its looping: progress is going to 100, then starting over.

I set up a 1/second loop using:  
while /bin/true; do date; curl -XGET  
'[http://192.168.1.5:9200/twitter/\_status?pretty=true](http://192.168.1.5:9200/twitter/_status?pretty=true)'; sleep 1; done  
then copied it to gist at: [540711’s gists · GitHub](http://gist.github.com/540711)

It should have recovered by now, I would think.

...Ken

Shay Banon wrote:

> great, ping me if it does not end, I am here to help (we can make it  
> more interactive on IRC).
> 
> p.s. Can you keep the original json format when you gist? Much easier to  
> know whats going on. You can add pretty=true as a parameter to get it  
> pretty printed.
> 
> -shay.banon
> 
> On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman  
> \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> ```
> I think so... Here's the latest on gist http://gist.github.com/540471
> 
> Thanks for the pointer on gist, I've never used it before.
> 
> Shay Banon wrote:
> > The top just states which shards were queries, a shard that is
> still not
> > allocated will obviously not be allocated. It seems like its still in
> > recovery process. There are two main APIs to really understand what is
> > going on (except for the high level health api), the cluster state
> API,
> > that shows you what the cluster wide state is (where each shard is
> > supposed to be, what its state is), and the status api which gives you
> > detailed information of the status of each shard allocated on each
> node.
> >
> > Is the recovery progressing?
> >
> > p.s. Can you use gist instead of pastebin?
> >
> > -shay.banon
> >
> > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > I restarted and now 35 of 36 are successful, but if you look
> at the
> > status, it's showing multiple shards in recovery. I'm confused.
> >
> > See cluster status in http://pastebin.com/9qWLf3mk
> >
> > Kenneth Loafman wrote:
> > > Will do so in just a bit...
> > >
> > > Shay Banon wrote:
> > >> ... can you test?
> > >>
> > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > >> <shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>> wrote:
> > >>
> > >> Just pushed a fix for this.
> > >>
> > >>
> > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >>
> > >> Attachments did not make it. See:
> > >> http://pastebin.com/ziALRgx5 -- cluster state
> > >> http://pastebin.com/63Xm95xM -- index status
> > >>
> > >> Sorry, they lost their formatting on Pastbin.
> > >>
> > >> ...Ken
> > >>
> > >> Kenneth Loafman wrote:
> > >> > I upgraded to last nights version, restarted, and
> > things are
> > >> worse. Now
> > >> > I have 5 shards hung at recover, not all on the same
> > node. Weird.
> > >> >
> > >> > I've attached the info you want. I'll leave
> things running
> > >> for now.
> > >> >
> > >> > ...Thanks,
> > >> > ...Ken
> > >> >
> > >> > Shay Banon wrote:
> > >> >> Do you still have it running? Can you gist the
> cluster
> > state
> > >> and the
> > >> >> index status results?
> > >> >>
> > >> >> I see that you are using master, I have fixed
> several
> > things
> > >> in this
> > >> >> area, can you pull a new version?
> > >> >>
> > >> >> -shay.banon
> > >> >>
> > >> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> > >> >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > >> >>
> > >> >> It seems to have started recover, but it's been
> > 7.5 hours
> > >> and appears to
> > >> >> be stopped/hung...
> > >> >>
> > >> >> > "1": [
> > >> >> > {
> > >> >> > "gateway_recovery": {
> > >> >> > "index": {
> > >> >> >
> > >> "expected_recovered_size": "0b",
> > >> >> >
> > >> "expected_recovered_size_in_bytes": 0,
> > >> >> >
> > "recovered_size": "0b",
> > >> >> >
> > >> "recovered_size_in_bytes": 0,
> > >> >> >                                
> "reused_size": "0b",
> > >> >> >
> > "reused_size_in_bytes": 0,
> > >> >> > "size": "0b",
> > >> >> >                                
> "size_in_bytes": 0,
> > >> >> >
> > "throttling_time": "0s",
> > >> >> >
> > >> "throttling_time_in_millis": 0
> > >> >> > },
> > >> >> > "stage": "RETRY",
> > >> >> >                            
> "start_time_in_millis":
> > >> 1282226019603,
> > >> >> > "throttling_time":
> > "7.6h",
> > >> >> >
> > >> "throttling_time_in_millis": 27514627,
> > >> >> > "time": "7.6h",
> > >> >> > "time_in_millis":
> > 27514657,
> > >> >> > "translog": {
> > >> >> > "recovered": 0
> > >> >> > }
> > >> >> > },
> > >> >> > "index": {
> > >> >> > "size": "0b",
> > >> >> > "size_in_bytes": 0
> > >> >> > },
> > >> >> > "routing": {
> > >> >> > "index":
> "twitter",
> > >> >> > "node":
> > >> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > >> >> > "primary": true,
> > >> >> >                            
> "relocating_node": null,
> > >> >> > "shard": 1,
> > >> >> > "state":
> "INITIALIZING"
> > >> >> > },
> > >> >> > "state": "RECOVERING"
> > >> >> > }
> > >> >> > ]
> > >> >>
> > >> >>
> > >> >> Shay Banon wrote:
> > >> >> > It should be allocated on the other node, you
> > shouldn't
> > >> need to start
> > >> >> > another node. When you issue a cluster health
> > (simple
> > >> curl can
> > >> >> do), what
> > >> >> > is the status? The cluster state API gives
> you more
> > >> information if you
> > >> >> > are after (each shard and its state).
> > >> >> >
> > >> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth
> Loafman
> > >> >> > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > >> >> >
> > >> >> > No this is the first time. The
> shutdown took a
> > >> while with several
> > >> >> > 'Waiting for not to shutdown..." style
> > message. It
> > >> came up bad
> > >> >> > after that.
> > >> >> >
> > >> >> > So, if I have two nodes now, and one
> needs to be
> > >> recovered,
> > >> >> I'll need 3
> > >> >> > nodes to get the recovery done?
> > >> >> >
> > >> >> > ...Ken
> > >> >> >
> > >> >> > Shay Banon wrote:
> > >> >> > > The shard will allocated to another
> node and
> > >> recovered there. Do
> > >> >> > you see
> > >> >> > > it happen continuously?
> > >> >> > >
> > >> >> > > -shay.banon
> > >> >> > >
> > >> >> > > On Thu, Aug 19, 2010 at 2:28 PM, Kenneth
> > Loafman
> > >> >> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > >> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > >> >> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > >> >> > >
> > >> >> > > Hi,
> > >> >> > >
> > >> >> > > The second shard on one of my
> indexes has
> > >> failed due to:
> > >> >> > > [05:59:47,332][WARN][index.gateway
> > >> ] [Mangog]
> > >> >> > [twitter][1]
> > >> >> > > failed to snapshot on close
> > >> >> > > ...followed by a long traceback.
> > >> >> > > ...followed by:
> > >> >> > > [05:59:49,336][WARN
> ][cluster.action.shard
> > >> ] [Mangog]
> > >> >> > received shard
> > >> >> > > failed for [twitter][1],
> > >> >> >    
> node[86d601df-e124-45ed-a5f2-57d762042d87],
> > >> >> > > [P], s[INITIALIZING], reason [Failed
> > to start
> > >> shard, message
> > >> >> > >
> > >> [IndexShardGatewayRecoveryException[[twitter][1]
> Failed to
> > >> >> > recovery
> > >> >> > > translog]; nested:
> > >> >> EngineCreationFailureException[[twitter][1]
> > >> >> > Failed to
> > >> >> > > open reader on writer]; nested:
> > >> >> > >
> > >> >> >
> > >> >>
> > >>
> >    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > >> >> > > (No such file or directory)]; ]]
> > >> >> > >
> > >> >> > > Is the recovery process
> automatic, or do I
> > >> have to do
> > >> >> something
> > >> >> > > special? It appears to be just this
> > one shard.
> > >> >> > >
> > >> >> > > I use the service wrapper to
> start/stop
> > >> 0.9.1-SNAPSHOT,
> > >> >> and my
> > >> >> > config is
> > >> >> > > below.
> > >> >> > >
> > >> >> > > ...Thanks,
> > >> >> > > ...Ken
> > >> >> > >
> > >> >> > > cloud:
> > >> >> > > aws:
> > >> >> > > access_key: *****
> > >> >> > > secret_key: *****
> > >> >> > >
> > >> >> > > gateway:
> > >> >> > > type: s3
> > >> >> > > s3:
> > >> >> > > bucket: *****
> > >> >> > >
> > >> >> > > path :
> > >> >> > > work : /mnt/search-data-dev
> > >> >> > > logs :
> /mnt/search-data-dev/node1/logs
> > >> >> > >
> > >> >> > > index :
> > >> >> > > number_of_shards : 2
> > >> >> > > number_of_replicas : 1
> > >> >> > >
> > >> >> > > network :
> > >> >> > > host : 192.168.1.5
> > >> >> > >
> > >> >> > >
> > >> >> >
> > >> >> >
> > >> >>
> > >> >>
> > >> >
> > >>
> > >>
> > >>
> > >
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 5:04pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/17 "2010-08-20T17:04:33Z")

</div>

Do you see any exceptions in the logs (failing to start the shard)?

On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Now its looping: progress is going to 100, then starting over.
> 
> I set up a 1/second loop using:  
> while /bin/true; do date; curl -XGET  
> '[http://192.168.1.5:9200/twitter/\_status?pretty=true](http://192.168.1.5:9200/twitter/_status?pretty=true)'; sleep 1; done  
> then copied it to gist at: [540711’s gists · GitHub](http://gist.github.com/540711)
> 
> It should have recovered by now, I would think.
> 
> ...Ken
> 
> Shay Banon wrote:
> 
> > great, ping me if it does not end, I am here to help (we can make it  
> > more interactive on IRC).
> > 
> > p.s. Can you keep the original json format when you gist? Much easier to  
> > know whats going on. You can add pretty=true as a parameter to get it  
> > pretty printed.
> > 
> > -shay.banon
> > 
> > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman  
> > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > 
> > ```
> > I think so... Here's the latest on gist
> > 
> > ```
> 
> [index status · GitHub](http://gist.github.com/540471)
> 
> > ```
> > Thanks for the pointer on gist, I've never used it before.
> > 
> > Shay Banon wrote:
> > > The top just states which shards were queries, a shard that is
> > still not
> > > allocated will obviously not be allocated. It seems like its still
> > 
> > ```
> 
> in
> 
> > ```
> > > recovery process. There are two main APIs to really understand what
> > 
> > ```
> 
> is
> 
> > ```
> > > going on (except for the high level health api), the cluster state
> > API,
> > > that shows you what the cluster wide state is (where each shard is
> > > supposed to be, what its state is), and the status api which gives
> > 
> > ```
> 
> you
> 
> > ```
> > > detailed information of the status of each shard allocated on each
> > node.
> > >
> > > Is the recovery progressing?
> > >
> > > p.s. Can you use gist instead of pastebin?
> > >
> > > -shay.banon
> > >
> > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >
> > > I restarted and now 35 of 36 are successful, but if you look
> > at the
> > > status, it's showing multiple shards in recovery. I'm
> > 
> > ```
> 
> confused.
> 
> > ```
> > >
> > > See cluster status in http://pastebin.com/9qWLf3mk
> > >
> > > Kenneth Loafman wrote:
> > > > Will do so in just a bit...
> > > >
> > > > Shay Banon wrote:
> > > >> ... can you test?
> > > >>
> > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > >> <shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>>> wrote:
> > > >>
> > > >> Just pushed a fix for this.
> > > >>
> > > >>
> > > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > > >> <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > >>
> > > >> Attachments did not make it. See:
> > > >> http://pastebin.com/ziALRgx5 -- cluster state
> > > >> http://pastebin.com/63Xm95xM -- index status
> > > >>
> > > >> Sorry, they lost their formatting on Pastbin.
> > > >>
> > > >> ...Ken
> > > >>
> > > >> Kenneth Loafman wrote:
> > > >> > I upgraded to last nights version, restarted, and
> > > things are
> > > >> worse. Now
> > > >> > I have 5 shards hung at recover, not all on the
> > 
> > ```
> 
> same
> 
> > ```
> > > node. Weird.
> > > >> >
> > > >> > I've attached the info you want. I'll leave
> > things running
> > > >> for now.
> > > >> >
> > > >> > ...Thanks,
> > > >> > ...Ken
> > > >> >
> > > >> > Shay Banon wrote:
> > > >> >> Do you still have it running? Can you gist the
> > cluster
> > > state
> > > >> and the
> > > >> >> index status results?
> > > >> >>
> > > >> >> I see that you are using master, I have fixed
> > several
> > > things
> > > >> in this
> > > >> >> area, can you pull a new version?
> > > >> >>
> > > >> >> -shay.banon
> > > >> >>
> > > >> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth Loafman
> > > >> >> <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > >> >>
> > > >> >> It seems to have started recover, but it's
> > 
> > ```
> 
> been
> 
> > ```
> > > 7.5 hours
> > > >> and appears to
> > > >> >> be stopped/hung...
> > > >> >>
> > > >> >> > "1": [
> > > >> >> > {
> > > >> >> > "gateway_recovery":
> > 
> > ```
> 
> {
> 
> > ```
> > > >> >> > "index": {
> > > >> >> >
> > > >> "expected_recovered_size": "0b",
> > > >> >> >
> > > >> "expected_recovered_size_in_bytes": 0,
> > > >> >> >
> > > "recovered_size": "0b",
> > > >> >> >
> > > >> "recovered_size_in_bytes": 0,
> > > >> >> >
> > "reused_size": "0b",
> > > >> >> >
> > > "reused_size_in_bytes": 0,
> > > >> >> > "size":
> > 
> > ```
> 
> "0b",
> 
> > ```
> > > >> >> >
> > "size_in_bytes": 0,
> > > >> >> >
> > > "throttling_time": "0s",
> > > >> >> >
> > > >> "throttling_time_in_millis": 0
> > > >> >> > },
> > > >> >> > "stage":
> > 
> > ```
> 
> "RETRY",
> 
> > ```
> > > >> >> >
> > "start_time_in_millis":
> > > >> 1282226019603,
> > > >> >> >
> > 
> > ```
> 
> "throttling\_time":
> 
> > ```
> > > "7.6h",
> > > >> >> >
> > > >> "throttling_time_in_millis": 27514627,
> > > >> >> > "time": "7.6h",
> > > >> >> >
> > 
> > ```
> 
> "time\_in\_millis":
> 
> > ```
> > > 27514657,
> > > >> >> > "translog": {
> > > >> >> >
> > 
> > ```
> 
> "recovered": 0
> 
> > ```
> > > >> >> > }
> > > >> >> > },
> > > >> >> > "index": {
> > > >> >> > "size": "0b",
> > > >> >> >
> > 
> > ```
> 
> "size\_in\_bytes": 0
> 
> > ```
> > > >> >> > },
> > > >> >> > "routing": {
> > > >> >> > "index":
> > "twitter",
> > > >> >> > "node":
> > > >> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > >> >> > "primary":
> > 
> > ```
> 
> true,
> 
> > ```
> > > >> >> >
> > "relocating_node": null,
> > > >> >> > "shard": 1,
> > > >> >> > "state":
> > "INITIALIZING"
> > > >> >> > },
> > > >> >> > "state":
> > 
> > ```
> 
> "RECOVERING"
> 
> > ```
> > > >> >> > }
> > > >> >> > ]
> > > >> >>
> > > >> >>
> > > >> >> Shay Banon wrote:
> > > >> >> > It should be allocated on the other node,
> > 
> > ```
> 
> you
> 
> > ```
> > > shouldn't
> > > >> need to start
> > > >> >> > another node. When you issue a cluster
> > 
> > ```
> 
> health
> 
> > ```
> > > (simple
> > > >> curl can
> > > >> >> do), what
> > > >> >> > is the status? The cluster state API gives
> > you more
> > > >> information if you
> > > >> >> > are after (each shard and its state).
> > > >> >> >
> > > >> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth
> > Loafman
> > > >> >> > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > > >> >> >
> > > >> >> > No this is the first time. The
> > shutdown took a
> > > >> while with several
> > > >> >> > 'Waiting for not to shutdown..." style
> > > message. It
> > > >> came up bad
> > > >> >> > after that.
> > > >> >> >
> > > >> >> > So, if I have two nodes now, and one
> > needs to be
> > > >> recovered,
> > > >> >> I'll need 3
> > > >> >> > nodes to get the recovery done?
> > > >> >> >
> > > >> >> > ...Ken
> > > >> >> >
> > > >> >> > Shay Banon wrote:
> > > >> >> > > The shard will allocated to another
> > node and
> > > >> recovered there. Do
> > > >> >> > you see
> > > >> >> > > it happen continuously?
> > > >> >> > >
> > > >> >> > > -shay.banon
> > > >> >> > >
> > > >> >> > > On Thu, Aug 19, 2010 at 2:28 PM,
> > 
> > ```
> 
> Kenneth
> 
> > ```
> > > Loafman
> > > >> >> > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>
> > > >> >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > > >> >> > >
> > > >> >> > > Hi,
> > > >> >> > >
> > > >> >> > > The second shard on one of my
> > indexes has
> > > >> failed due to:
> > > >> >> > > [05:59:47,332][WARN
> > 
> > ```
> 
> ][index.gateway
> 
> > ```
> > > >> ] [Mangog]
> > > >> >> > [twitter][1]
> > > >> >> > > failed to snapshot on close
> > > >> >> > > ...followed by a long traceback.
> > > >> >> > > ...followed by:
> > > >> >> > > [05:59:49,336][WARN
> > ][cluster.action.shard
> > > >> ] [Mangog]
> > > >> >> > received shard
> > > >> >> > > failed for [twitter][1],
> > > >> >> >
> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > >> >> > > [P], s[INITIALIZING], reason
> > 
> > ```
> 
> [Failed
> 
> > ```
> > > to start
> > > >> shard, message
> > > >> >> > >
> > > >> [IndexShardGatewayRecoveryException[[twitter][1]
> > Failed to
> > > >> >> > recovery
> > > >> >> > > translog]; nested:
> > > >> >> EngineCreationFailureException[[twitter][1]
> > > >> >> > Failed to
> > > >> >> > > open reader on writer]; nested:
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> > 
> > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > ```
> > > >> >> > > (No such file or directory)]; ]]
> > > >> >> > >
> > > >> >> > > Is the recovery process
> > automatic, or do I
> > > >> have to do
> > > >> >> something
> > > >> >> > > special? It appears to be just
> > 
> > ```
> 
> this
> 
> > ```
> > > one shard.
> > > >> >> > >
> > > >> >> > > I use the service wrapper to
> > start/stop
> > > >> 0.9.1-SNAPSHOT,
> > > >> >> and my
> > > >> >> > config is
> > > >> >> > > below.
> > > >> >> > >
> > > >> >> > > ...Thanks,
> > > >> >> > > ...Ken
> > > >> >> > >
> > > >> >> > > cloud:
> > > >> >> > > aws:
> > > >> >> > > access_key: *****
> > > >> >> > > secret_key: *****
> > > >> >> > >
> > > >> >> > > gateway:
> > > >> >> > > type: s3
> > > >> >> > > s3:
> > > >> >> > > bucket: *****
> > > >> >> > >
> > > >> >> > > path :
> > > >> >> > > work : /mnt/search-data-dev
> > > >> >> > > logs :
> > /mnt/search-data-dev/node1/logs
> > > >> >> > >
> > > >> >> > > index :
> > > >> >> > > number_of_shards : 2
> > > >> >> > > number_of_replicas : 1
> > > >> >> > >
> > > >> >> > > network :
> > > >> >> > > host : 192.168.1.5
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> > 
> > ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 5:05pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/18 "2010-08-20T17:05:01Z")

</div>

Also, use the latest again, pushed some more fixes.

On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote:

> Do you see any exceptions in the logs (failing to start the shard)?
> 
> On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman \<  
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> 
> > Now its looping: progress is going to 100, then starting over.
> > 
> > I set up a 1/second loop using:  
> > while /bin/true; do date; curl -XGET  
> > '[http://192.168.1.5:9200/twitter/\_status?pretty=true](http://192.168.1.5:9200/twitter/_status?pretty=true)'; sleep 1; done  
> > then copied it to gist at: [540711’s gists · GitHub](http://gist.github.com/540711)
> > 
> > It should have recovered by now, I would think.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > 
> > > great, ping me if it does not end, I am here to help (we can make it  
> > > more interactive on IRC).
> > > 
> > > p.s. Can you keep the original json format when you gist? Much easier to  
> > > know whats going on. You can add pretty=true as a parameter to get it  
> > > pretty printed.
> > > 
> > > -shay.banon
> > > 
> > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman  
> > > \<[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com) [mailto:kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\> wrote:
> > > 
> > > ```
> > > I think so... Here's the latest on gist
> > > 
> > > ```
> > 
> > [index status · GitHub](http://gist.github.com/540471)
> > 
> > > ```
> > > Thanks for the pointer on gist, I've never used it before.
> > > 
> > > Shay Banon wrote:
> > > > The top just states which shards were queries, a shard that is
> > > still not
> > > > allocated will obviously not be allocated. It seems like its still
> > > 
> > > ```
> > 
> > in
> > 
> > > ```
> > > > recovery process. There are two main APIs to really understand
> > > 
> > > ```
> > 
> > what is
> > 
> > > ```
> > > > going on (except for the high level health api), the cluster state
> > > API,
> > > > that shows you what the cluster wide state is (where each shard is
> > > > supposed to be, what its state is), and the status api which gives
> > > 
> > > ```
> > 
> > you
> > 
> > > ```
> > > > detailed information of the status of each shard allocated on each
> > > node.
> > > >
> > > > Is the recovery progressing?
> > > >
> > > > p.s. Can you use gist instead of pastebin?
> > > >
> > > > -shay.banon
> > > >
> > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > > >
> > > > I restarted and now 35 of 36 are successful, but if you look
> > > at the
> > > > status, it's showing multiple shards in recovery. I'm
> > > 
> > > ```
> > 
> > confused.
> > 
> > > ```
> > > >
> > > > See cluster status in http://pastebin.com/9qWLf3mk
> > > >
> > > > Kenneth Loafman wrote:
> > > > > Will do so in just a bit...
> > > > >
> > > > > Shay Banon wrote:
> > > > >> ... can you test?
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > > >> <shay.banon@elasticsearch.com
> > > <mailto:shay.banon@elasticsearch.com>
> > > > <mailto:shay.banon@elasticsearch.com
> > > <mailto:shay.banon@elasticsearch.com>>
> > > > <mailto:shay.banon@elasticsearch.com
> > > <mailto:shay.banon@elasticsearch.com>
> > > > <mailto:shay.banon@elasticsearch.com
> > > <mailto:shay.banon@elasticsearch.com>>>> wrote:
> > > > >>
> > > > >> Just pushed a fix for this.
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > > > >> <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com> <mailto:
> > > 
> > > ```
> > 
> > [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)
> > 
> > > ```
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > > >>
> > > > >> Attachments did not make it. See:
> > > > >> http://pastebin.com/ziALRgx5 -- cluster state
> > > > >> http://pastebin.com/63Xm95xM -- index status
> > > > >>
> > > > >> Sorry, they lost their formatting on Pastbin.
> > > > >>
> > > > >> ...Ken
> > > > >>
> > > > >> Kenneth Loafman wrote:
> > > > >> > I upgraded to last nights version, restarted, and
> > > > things are
> > > > >> worse. Now
> > > > >> > I have 5 shards hung at recover, not all on the
> > > 
> > > ```
> > 
> > same
> > 
> > > ```
> > > > node. Weird.
> > > > >> >
> > > > >> > I've attached the info you want. I'll leave
> > > things running
> > > > >> for now.
> > > > >> >
> > > > >> > ...Thanks,
> > > > >> > ...Ken
> > > > >> >
> > > > >> > Shay Banon wrote:
> > > > >> >> Do you still have it running? Can you gist the
> > > cluster
> > > > state
> > > > >> and the
> > > > >> >> index status results?
> > > > >> >>
> > > > >> >> I see that you are using master, I have fixed
> > > several
> > > > things
> > > > >> in this
> > > > >> >> area, can you pull a new version?
> > > > >> >>
> > > > >> >> -shay.banon
> > > > >> >>
> > > > >> >> On Fri, Aug 20, 2010 at 12:33 AM, Kenneth
> > > 
> > > ```
> > 
> > Loafman
> > 
> > > ```
> > > > >> >> <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com
> > > 
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > > >> >>
> > > > >> >> It seems to have started recover, but it's
> > > 
> > > ```
> > 
> > been
> > 
> > > ```
> > > > 7.5 hours
> > > > >> and appears to
> > > > >> >> be stopped/hung...
> > > > >> >>
> > > > >> >> > "1": [
> > > > >> >> > {
> > > > >> >> >
> > > 
> > > ```
> > 
> > "gateway\_recovery": {
> > 
> > > ```
> > > > >> >> > "index": {
> > > > >> >> >
> > > > >> "expected_recovered_size": "0b",
> > > > >> >> >
> > > > >> "expected_recovered_size_in_bytes": 0,
> > > > >> >> >
> > > > "recovered_size": "0b",
> > > > >> >> >
> > > > >> "recovered_size_in_bytes": 0,
> > > > >> >> >
> > > "reused_size": "0b",
> > > > >> >> >
> > > > "reused_size_in_bytes": 0,
> > > > >> >> > "size":
> > > 
> > > ```
> > 
> > "0b",
> > 
> > > ```
> > > > >> >> >
> > > "size_in_bytes": 0,
> > > > >> >> >
> > > > "throttling_time": "0s",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 0
> > > > >> >> > },
> > > > >> >> > "stage":
> > > 
> > > ```
> > 
> > "RETRY",
> > 
> > > ```
> > > > >> >> >
> > > "start_time_in_millis":
> > > > >> 1282226019603,
> > > > >> >> >
> > > 
> > > ```
> > 
> > "throttling\_time":
> > 
> > > ```
> > > > "7.6h",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 27514627,
> > > > >> >> > "time":
> > > 
> > > ```
> > 
> > "7.6h",
> > 
> > > ```
> > > > >> >> >
> > > 
> > > ```
> > 
> > "time\_in\_millis":
> > 
> > > ```
> > > > 27514657,
> > > > >> >> > "translog": {
> > > > >> >> >
> > > 
> > > ```
> > 
> > "recovered": 0
> > 
> > > ```
> > > > >> >> > }
> > > > >> >> > },
> > > > >> >> > "index": {
> > > > >> >> > "size": "0b",
> > > > >> >> >
> > > 
> > > ```
> > 
> > "size\_in\_bytes": 0
> > 
> > > ```
> > > > >> >> > },
> > > > >> >> > "routing": {
> > > > >> >> > "index":
> > > "twitter",
> > > > >> >> > "node":
> > > > >> >> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > > >> >> > "primary":
> > > 
> > > ```
> > 
> > true,
> > 
> > > ```
> > > > >> >> >
> > > "relocating_node": null,
> > > > >> >> > "shard": 1,
> > > > >> >> > "state":
> > > "INITIALIZING"
> > > > >> >> > },
> > > > >> >> > "state":
> > > 
> > > ```
> > 
> > "RECOVERING"
> > 
> > > ```
> > > > >> >> > }
> > > > >> >> > ]
> > > > >> >>
> > > > >> >>
> > > > >> >> Shay Banon wrote:
> > > > >> >> > It should be allocated on the other node,
> > > 
> > > ```
> > 
> > you
> > 
> > > ```
> > > > shouldn't
> > > > >> need to start
> > > > >> >> > another node. When you issue a cluster
> > > 
> > > ```
> > 
> > health
> > 
> > > ```
> > > > (simple
> > > > >> curl can
> > > > >> >> do), what
> > > > >> >> > is the status? The cluster state API gives
> > > you more
> > > > >> information if you
> > > > >> >> > are after (each shard and its state).
> > > > >> >> >
> > > > >> >> > On Thu, Aug 19, 2010 at 3:48 PM, Kenneth
> > > Loafman
> > > > >> >> > <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > > > >> >> >
> > > > >> >> > No this is the first time. The
> > > shutdown took a
> > > > >> while with several
> > > > >> >> > 'Waiting for not to shutdown..." style
> > > > message. It
> > > > >> came up bad
> > > > >> >> > after that.
> > > > >> >> >
> > > > >> >> > So, if I have two nodes now, and one
> > > needs to be
> > > > >> recovered,
> > > > >> >> I'll need 3
> > > > >> >> > nodes to get the recovery done?
> > > > >> >> >
> > > > >> >> > ...Ken
> > > > >> >> >
> > > > >> >> > Shay Banon wrote:
> > > > >> >> > > The shard will allocated to another
> > > node and
> > > > >> recovered there. Do
> > > > >> >> > you see
> > > > >> >> > > it happen continuously?
> > > > >> >> > >
> > > > >> >> > > -shay.banon
> > > > >> >> > >
> > > > >> >> > > On Thu, Aug 19, 2010 at 2:28 PM,
> > > 
> > > ```
> > 
> > Kenneth
> > 
> > > ```
> > > > Loafman
> > > > >> >> > > <kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com
> > > 
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >> > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>
> > > > <mailto:kenneth.loafman@gmail.com
> > > <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > > > >> >> > >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > > The second shard on one of my
> > > indexes has
> > > > >> failed due to:
> > > > >> >> > > [05:59:47,332][WARN
> > > 
> > > ```
> > 
> > ][index.gateway
> > 
> > > ```
> > > > >> ] [Mangog]
> > > > >> >> > [twitter][1]
> > > > >> >> > > failed to snapshot on close
> > > > >> >> > > ...followed by a long traceback.
> > > > >> >> > > ...followed by:
> > > > >> >> > > [05:59:49,336][WARN
> > > ][cluster.action.shard
> > > > >> ] [Mangog]
> > > > >> >> > received shard
> > > > >> >> > > failed for [twitter][1],
> > > > >> >> >
> > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > >> >> > > [P], s[INITIALIZING], reason
> > > 
> > > ```
> > 
> > [Failed
> > 
> > > ```
> > > > to start
> > > > >> shard, message
> > > > >> >> > >
> > > > >> [IndexShardGatewayRecoveryException[[twitter][1]
> > > Failed to
> > > > >> >> > recovery
> > > > >> >> > > translog]; nested:
> > > > >> >> EngineCreationFailureException[[twitter][1]
> > > > >> >> > Failed to
> > > > >> >> > > open reader on writer]; nested:
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > > 
> > > ```
> > 
> > FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> > 
> > > ```
> > > > >> >> > > (No such file or directory)]; ]]
> > > > >> >> > >
> > > > >> >> > > Is the recovery process
> > > automatic, or do I
> > > > >> have to do
> > > > >> >> something
> > > > >> >> > > special? It appears to be just
> > > 
> > > ```
> > 
> > this
> > 
> > > ```
> > > > one shard.
> > > > >> >> > >
> > > > >> >> > > I use the service wrapper to
> > > start/stop
> > > > >> 0.9.1-SNAPSHOT,
> > > > >> >> and my
> > > > >> >> > config is
> > > > >> >> > > below.
> > > > >> >> > >
> > > > >> >> > > ...Thanks,
> > > > >> >> > > ...Ken
> > > > >> >> > >
> > > > >> >> > > cloud:
> > > > >> >> > > aws:
> > > > >> >> > > access_key: *****
> > > > >> >> > > secret_key: *****
> > > > >> >> > >
> > > > >> >> > > gateway:
> > > > >> >> > > type: s3
> > > > >> >> > > s3:
> > > > >> >> > > bucket: *****
> > > > >> >> > >
> > > > >> >> > > path :
> > > > >> >> > > work : /mnt/search-data-dev
> > > > >> >> > > logs :
> > > /mnt/search-data-dev/node1/logs
> > > > >> >> > >
> > > > >> >> > > index :
> > > > >> >> > > number_of_shards : 2
> > > > >> >> > > number_of_replicas : 1
> > > > >> >> > >
> > > > >> >> > > network :
> > > > >> >> > > host : 192.168.1.5
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > > 
> > > ```

---

<div class="post-metadata">

**Author:** ![Kenneth\_Loafman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenneth_loafman/32/2224_2.png) [@Kenneth\_Loafman](https://discuss.elastic.co/u/Kenneth_Loafman)\
**Post date:** [August 20, 2010, 5:13pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/19 "2010-08-20T17:13:49Z")

</div>

Looks like a file may be missing on the gateway... this repeats in the  
log over and over.

[12:10:00,597][WARN][indices.cluster] [Magilla] [twitter][1]  
failed to start shard  
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
[twitter][1] Failed to recover translog  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:516)  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:417)  
at  
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:172)  
at  
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
at  
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
at java.lang.Thread.run(Thread.java:636)  
Caused by:  
org.elasticsearch.index.engine.EngineCreationFailureException:  
[twitter][1] Failed to open reader on writer  
at  
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171)  
at  
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:405)  
at  
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:440)  
... 5 more  
Caused by: java.io.FileNotFoundException:  
/mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/\_d8g.cfs  
(No such file or directory)  
at java.io.RandomAccessFile.open(Native Method)  
at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
at  
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:76)  
at  
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:97)  
at  
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:87)  
at  
org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)  
at  
org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInput(AbstractStore.java:287)  
at  
org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:67)  
at  
org.apache.lucene.index.SegmentReader$CoreReaders.(SegmentReader.java:114)  
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)  
at  
org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)  
at  
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:574)  
at  
org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:150)  
at  
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)  
at  
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)  
at  
org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngine.java:538)  
at  
org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158)  
... 7 more  
[12:10:00,605][WARN][cluster.action.shard] [Magilla] sending  
failed shard for [twitter][1],  
node[10dab323-019b-4036-854f-89bb068dcc8d], [P], s[INITIALIZING], reason  
[Failed to start shard, message  
[IndexShardGatewayRecoveryException[[twitter][1] Failed to recover  
translog]; nested: EngineCreationFailureException[[twitter][1] Failed to  
open reader on writer]; nested:  
FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/\_d8g.cfs  
(No such file or directory)]; ]]

Shay Banon wrote:

> Also, use the latest again, pushed some more fixes.
> 
> On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon  
> \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) [mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\> wrote:
> 
> ```
> Do you see any exceptions in the logs (failing to start the shard)?
> 
> On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>> wrote:
> 
> Now its looping: progress is going to 100, then starting over.
> 
> I set up a 1/second loop using:
> while /bin/true; do date; curl -XGET
> 'http://192.168.1.5:9200/twitter/_status?pretty=true'; sleep 1; done
> then copied it to gist at: http://gist.github.com/540711
> 
> It should have recovered by now, I would think.
> 
> ...Ken
> 
> Shay Banon wrote:
> > great, ping me if it does not end, I am here to help (we can
> make it
> > more interactive on IRC).
> >
> > p.s. Can you keep the original json format when you gist? Much
> easier to
> > know whats going on. You can add pretty=true as a parameter to
> get it
> > pretty printed.
> >
> > -shay.banon
> >
> > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>> wrote:
> >
> > I think so... Here's the latest on gist
> http://gist.github.com/540471
> >
> > Thanks for the pointer on gist, I've never used it before.
> >
> > Shay Banon wrote:
> > > The top just states which shards were queries, a shard
> that is
> > still not
> > > allocated will obviously not be allocated. It seems like
> its still in
> > > recovery process. There are two main APIs to really
> understand what is
> > > going on (except for the high level health api), the
> cluster state
> > API,
> > > that shows you what the cluster wide state is (where
> each shard is
> > > supposed to be, what its state is), and the status api
> which gives you
> > > detailed information of the status of each shard
> allocated on each
> > node.
> > >
> > > Is the recovery progressing?
> > >
> > > p.s. Can you use gist instead of pastebin?
> > >
> > > -shay.banon
> > >
> > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > >
> > > I restarted and now 35 of 36 are successful, but if
> you look
> > at the
> > > status, it's showing multiple shards in recovery.
> I'm confused.
> > >
> > > See cluster status in http://pastebin.com/9qWLf3mk
> > >
> > > Kenneth Loafman wrote:
> > > > Will do so in just a bit...
> > > >
> > > > Shay Banon wrote:
> > > >> ... can you test?
> > > >>
> > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > >> <shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>
> > > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>
> > <mailto:shay.banon@elasticsearch.com
> <mailto:shay.banon@elasticsearch.com>>>>> wrote:
> > > >>
> > > >> Just pushed a fix for this.
> > > >>
> > > >>
> > > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth Loafman
> > > >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > >>
> > > >> Attachments did not make it. See:
> > > >> http://pastebin.com/ziALRgx5 -- cluster state
> > > >> http://pastebin.com/63Xm95xM -- index status
> > > >>
> > > >> Sorry, they lost their formatting on Pastbin.
> > > >>
> > > >> ...Ken
> > > >>
> > > >> Kenneth Loafman wrote:
> > > >> > I upgraded to last nights version,
> restarted, and
> > > things are
> > > >> worse. Now
> > > >> > I have 5 shards hung at recover, not
> all on the same
> > > node. Weird.
> > > >> >
> > > >> > I've attached the info you want. I'll
> leave
> > things running
> > > >> for now.
> > > >> >
> > > >> > ...Thanks,
> > > >> > ...Ken
> > > >> >
> > > >> > Shay Banon wrote:
> > > >> >> Do you still have it running? Can you
> gist the
> > cluster
> > > state
> > > >> and the
> > > >> >> index status results?
> > > >> >>
> > > >> >> I see that you are using master, I
> have fixed
> > several
> > > things
> > > >> in this
> > > >> >> area, can you pull a new version?
> > > >> >>
> > > >> >> -shay.banon
> > > >> >>
> > > >> >> On Fri, Aug 20, 2010 at 12:33 AM,
> Kenneth Loafman
> > > >> >> <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > > >> >>
> > > >> >> It seems to have started recover,
> but it's been
> > > 7.5 hours
> > > >> and appears to
> > > >> >> be stopped/hung...
> > > >> >>
> > > >> >> > "1": [
> > > >> >> > {
> > > >> >> >                        
> "gateway_recovery": {
> > > >> >> >                            
> "index": {
> > > >> >> >
> > > >> "expected_recovered_size": "0b",
> > > >> >> >
> > > >> "expected_recovered_size_in_bytes": 0,
> > > >> >> >
> > > "recovered_size": "0b",
> > > >> >> >
> > > >> "recovered_size_in_bytes": 0,
> > > >> >> >
> > "reused_size": "0b",
> > > >> >> >
> > > "reused_size_in_bytes": 0,
> > > >> >> >                                
> "size": "0b",
> > > >> >> >
> > "size_in_bytes": 0,
> > > >> >> >
> > > "throttling_time": "0s",
> > > >> >> >
> > > >> "throttling_time_in_millis": 0
> > > >> >> > },
> > > >> >> >                            
> "stage": "RETRY",
> > > >> >> >
> > "start_time_in_millis":
> > > >> 1282226019603,
> > > >> >> >                            
> "throttling_time":
> > > "7.6h",
> > > >> >> >
> > > >> "throttling_time_in_millis": 27514627,
> > > >> >> >                            
> "time": "7.6h",
> > > >> >> >                            
> "time_in_millis":
> > > 27514657,
> > > >> >> >                            
> "translog": {
> > > >> >> >                                
> "recovered": 0
> > > >> >> > }
> > > >> >> > },
> > > >> >> > "index": {
> > > >> >> >                            
> "size": "0b",
> > > >> >> >                            
> "size_in_bytes": 0
> > > >> >> > },
> > > >> >> > "routing": {
> > > >> >> > "index":
> > "twitter",
> > > >> >> > "node":
> > > >> >>    
> "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > >> >> >                            
> "primary": true,
> > > >> >> >
> > "relocating_node": null,
> > > >> >> >                            
> "shard": 1,
> > > >> >> > "state":
> > "INITIALIZING"
> > > >> >> > },
> > > >> >> > "state":
> "RECOVERING"
> > > >> >> > }
> > > >> >> > ]
> > > >> >>
> > > >> >>
> > > >> >> Shay Banon wrote:
> > > >> >> > It should be allocated on the
> other node, you
> > > shouldn't
> > > >> need to start
> > > >> >> > another node. When you issue a
> cluster health
> > > (simple
> > > >> curl can
> > > >> >> do), what
> > > >> >> > is the status? The cluster state
> API gives
> > you more
> > > >> information if you
> > > >> >> > are after (each shard and its
> state).
> > > >> >> >
> > > >> >> > On Thu, Aug 19, 2010 at 3:48 PM,
> Kenneth
> > Loafman
> > > >> >> > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > > >> >> >
> > > >> >> > No this is the first time. The
> > shutdown took a
> > > >> while with several
> > > >> >> > 'Waiting for not to
> shutdown..." style
> > > message. It
> > > >> came up bad
> > > >> >> > after that.
> > > >> >> >
> > > >> >> > So, if I have two nodes now,
> and one
> > needs to be
> > > >> recovered,
> > > >> >> I'll need 3
> > > >> >> > nodes to get the recovery done?
> > > >> >> >
> > > >> >> > ...Ken
> > > >> >> >
> > > >> >> > Shay Banon wrote:
> > > >> >> > > The shard will allocated
> to another
> > node and
> > > >> recovered there. Do
> > > >> >> > you see
> > > >> >> > > it happen continuously?
> > > >> >> > >
> > > >> >> > > -shay.banon
> > > >> >> > >
> > > >> >> > > On Thu, Aug 19, 2010 at
> 2:28 PM, Kenneth
> > > Loafman
> > > >> >> > > <kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>
> > > >> >> >    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>
> > > >> >> >    
> <mailto:kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>
> > > >> >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>
> > > >> <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> <mailto:kenneth.loafman@gmail.com>>>>>>>> wrote:
> > > >> >> > >
> > > >> >> > > Hi,
> > > >> >> > >
> > > >> >> > > The second shard on
> one of my
> > indexes has
> > > >> failed due to:
> > > >> >> > > [05:59:47,332][WARN
> ][index.gateway
> > > >> ] [Mangog]
> > > >> >> > [twitter][1]
> > > >> >> > > failed to snapshot on
> close
> > > >> >> > > ...followed by a long
> traceback.
> > > >> >> > > ...followed by:
> > > >> >> > > [05:59:49,336][WARN
> > ][cluster.action.shard
> > > >> ] [Mangog]
> > > >> >> > received shard
> > > >> >> > > failed for [twitter][1],
> > > >> >> >
> > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > >> >> > > [P], s[INITIALIZING],
> reason [Failed
> > > to start
> > > >> shard, message
> > > >> >> > >
> > > >>        
> [IndexShardGatewayRecoveryException[[twitter][1]
> > Failed to
> > > >> >> > recovery
> > > >> >> > > translog]; nested:
> > > >> >>    
> EngineCreationFailureException[[twitter][1]
> > > >> >> > Failed to
> > > >> >> > > open reader on
> writer]; nested:
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >>
> > >
> >    
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/_d8g.cfs
> > > >> >> > > (No such file or
> directory)]; ]]
> > > >> >> > >
> > > >> >> > > Is the recovery process
> > automatic, or do I
> > > >> have to do
> > > >> >> something
> > > >> >> > > special? It appears
> to be just this
> > > one shard.
> > > >> >> > >
> > > >> >> > > I use the service
> wrapper to
> > start/stop
> > > >> 0.9.1-SNAPSHOT,
> > > >> >> and my
> > > >> >> > config is
> > > >> >> > > below.
> > > >> >> > >
> > > >> >> > > ...Thanks,
> > > >> >> > > ...Ken
> > > >> >> > >
> > > >> >> > > cloud:
> > > >> >> > > aws:
> > > >> >> > > access_key: *****
> > > >> >> > > secret_key: *****
> > > >> >> > >
> > > >> >> > > gateway:
> > > >> >> > > type: s3
> > > >> >> > > s3:
> > > >> >> > > bucket: *****
> > > >> >> > >
> > > >> >> > > path :
> > > >> >> > > work :
> /mnt/search-data-dev
> > > >> >> > > logs :
> > /mnt/search-data-dev/node1/logs
> > > >> >> > >
> > > >> >> > > index :
> > > >> >> > > number_of_shards : 2
> > > >> >> > > number_of_replicas : 1
> > > >> >> > >
> > > >> >> > > network :
> > > >> >> > > host : 192.168.1.5
> > > >> >> > >
> > > >> >> > >
> > > >> >> >
> > > >> >> >
> > > >> >>
> > > >> >>
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
> >
> 
> ```

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [August 20, 2010, 5:23pm UTC](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239/20 "2010-08-20T17:23:47Z")

</div>

It means that the gateway store got corrupted. You will have to rebuild the  
index. Probably due to all HEAD changes... . Hopefully its getting stable  
now.

-shay.banon

On Fri, Aug 20, 2010 at 8:13 PM, Kenneth Loafman  
[kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)wrote:

> Looks like a file may be missing on the gateway... this repeats in the  
> log over and over.
> 
> [12:10:00,597][WARN][indices.cluster] [Magilla] [twitter][1]  
> failed to start shard  
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> [twitter][1] Failed to recover translog  
> at
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:516)  
> at
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:417)  
> at
> 
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:172)  
> at
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)  
> at
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)  
> at java.lang.Thread.run(Thread.java:636)  
> Caused by:  
> org.elasticsearch.index.engine.EngineCreationFailureException:  
> [twitter][1] Failed to open reader on writer  
> at
> 
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:171)  
> at
> 
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:405)  
> at
> 
> org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recoverTranslog(BlobStoreIndexShardGateway.java:440)  
> ... 5 more  
> Caused by: java.io.FileNotFoundException:  
> /mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/\_d8g.cfs  
> (No such file or directory)  
> at java.io.RandomAccessFile.open(Native Method)  
> at java.io.RandomAccessFile.(RandomAccessFile.java:233)  
> at
> 
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:76)  
> at
> 
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:97)  
> at
> 
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:87)  
> at  
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:67)  
> at
> 
> org.elasticsearch.index.store.support.AbstractStore$StoreDirectory.openInput(AbstractStore.java:287)  
> at
> 
> org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:67)  
> at
> 
> org.apache.lucene.index.SegmentReader$CoreReaders.(SegmentReader.java:114)  
> at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:590)  
> at  
> org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:616)  
> at
> 
> org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:574)  
> at  
> org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:150)  
> at
> 
> org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)  
> at  
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:410)  
> at  
> org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:374)  
> at
> 
> org.elasticsearch.index.engine.robin.RobinEngine.buildNrtResource(RobinEngine.java:538)  
> at
> 
> org.elasticsearch.index.engine.robin.RobinEngine.start(RobinEngine.java:158)  
> ... 7 more  
> [12:10:00,605][WARN][cluster.action.shard] [Magilla] sending  
> failed shard for [twitter][1],  
> node[10dab323-019b-4036-854f-89bb068dcc8d], [P], s[INITIALIZING], reason  
> [Failed to start shard, message  
> [IndexShardGatewayRecoveryException[[twitter][1] Failed to recover  
> translog]; nested: EngineCreationFailureException[[twitter][1] Failed to  
> open reader on writer]; nested:
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/1/indices/twitter/1/index/\_d8g.cfs  
> (No such file or directory)]; ]]
> 
> Shay Banon wrote:
> 
> > Also, use the latest again, pushed some more fixes.
> > 
> > On Fri, Aug 20, 2010 at 8:04 PM, Shay Banon  
> > \<[shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com) [mailto:shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)\>  
> > wrote:
> > 
> > ```
> > Do you see any exceptions in the logs (failing to start the shard)?
> > 
> > On Fri, Aug 20, 2010 at 8:02 PM, Kenneth Loafman
> > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>>
> > 
> > ```
> 
> wrote:
> 
> > ```
> > Now its looping: progress is going to 100, then starting over.
> > 
> > I set up a 1/second loop using:
> > while /bin/true; do date; curl -XGET
> > 'http://192.168.1.5:9200/twitter/_status?pretty=true'; sleep 1;
> > 
> > ```
> 
> done
> 
> > ```
> > then copied it to gist at: http://gist.github.com/540711
> > 
> > It should have recovered by now, I would think.
> > 
> > ...Ken
> > 
> > Shay Banon wrote:
> > > great, ping me if it does not end, I am here to help (we can
> > make it
> > > more interactive on IRC).
> > >
> > > p.s. Can you keep the original json format when you gist? Much
> > easier to
> > > know whats going on. You can add pretty=true as a parameter to
> > get it
> > > pretty printed.
> > >
> > > -shay.banon
> > >
> > > On Fri, Aug 20, 2010 at 5:51 PM, Kenneth Loafman
> > > <kenneth.loafman@gmail.com <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>> wrote:
> > >
> > > I think so... Here's the latest on gist
> > http://gist.github.com/540471
> > >
> > > Thanks for the pointer on gist, I've never used it before.
> > >
> > > Shay Banon wrote:
> > > > The top just states which shards were queries, a shard
> > that is
> > > still not
> > > > allocated will obviously not be allocated. It seems like
> > its still in
> > > > recovery process. There are two main APIs to really
> > understand what is
> > > > going on (except for the high level health api), the
> > cluster state
> > > API,
> > > > that shows you what the cluster wide state is (where
> > each shard is
> > > > supposed to be, what its state is), and the status api
> > which gives you
> > > > detailed information of the status of each shard
> > allocated on each
> > > node.
> > > >
> > > > Is the recovery progressing?
> > > >
> > > > p.s. Can you use gist instead of pastebin?
> > > >
> > > > -shay.banon
> > > >
> > > > On Fri, Aug 20, 2010 at 5:13 PM, Kenneth Loafman
> > > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>> wrote:
> > > >
> > > > I restarted and now 35 of 36 are successful, but if
> > you look
> > > at the
> > > > status, it's showing multiple shards in recovery.
> > I'm confused.
> > > >
> > > > See cluster status in http://pastebin.com/9qWLf3mk
> > > >
> > > > Kenneth Loafman wrote:
> > > > > Will do so in just a bit...
> > > > >
> > > > > Shay Banon wrote:
> > > > >> ... can you test?
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 4:02 PM, Shay Banon
> > > > >> <shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>
> > > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>>
> > > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>
> > > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>
> > > <mailto:shay.banon@elasticsearch.com
> > <mailto:shay.banon@elasticsearch.com>>>>> wrote:
> > > > >>
> > > > >> Just pushed a fix for this.
> > > > >>
> > > > >>
> > > > >> On Fri, Aug 20, 2010 at 3:31 PM, Kenneth
> > 
> > ```
> 
> Loafman
> 
> > ```
> > > > >> <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > <mailto:kenneth.loafman@gmail.com <mailto:
> > 
> > ```
> 
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\>
> 
> > ```
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>> wrote:
> > > > >>
> > > > >> Attachments did not make it. See:
> > > > >> http://pastebin.com/ziALRgx5 -- cluster
> > 
> > ```
> 
> state
> 
> > ```
> > > > >> http://pastebin.com/63Xm95xM -- index
> > 
> > ```
> 
> status
> 
> > ```
> > > > >>
> > > > >> Sorry, they lost their formatting on
> > 
> > ```
> 
> Pastbin.
> 
> > ```
> > > > >>
> > > > >> ...Ken
> > > > >>
> > > > >> Kenneth Loafman wrote:
> > > > >> > I upgraded to last nights version,
> > restarted, and
> > > > things are
> > > > >> worse. Now
> > > > >> > I have 5 shards hung at recover, not
> > all on the same
> > > > node. Weird.
> > > > >> >
> > > > >> > I've attached the info you want. I'll
> > leave
> > > things running
> > > > >> for now.
> > > > >> >
> > > > >> > ...Thanks,
> > > > >> > ...Ken
> > > > >> >
> > > > >> > Shay Banon wrote:
> > > > >> >> Do you still have it running? Can you
> > gist the
> > > cluster
> > > > state
> > > > >> and the
> > > > >> >> index status results?
> > > > >> >>
> > > > >> >> I see that you are using master, I
> > have fixed
> > > several
> > > > things
> > > > >> in this
> > > > >> >> area, can you pull a new version?
> > > > >> >>
> > > > >> >> -shay.banon
> > > > >> >>
> > > > >> >> On Fri, Aug 20, 2010 at 12:33 AM,
> > Kenneth Loafman
> > > > >> >> <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>> wrote:
> > > > >> >>
> > > > >> >> It seems to have started recover,
> > but it's been
> > > > 7.5 hours
> > > > >> and appears to
> > > > >> >> be stopped/hung...
> > > > >> >>
> > > > >> >> > "1": [
> > > > >> >> > {
> > > > >> >> >
> > "gateway_recovery": {
> > > > >> >> >
> > "index": {
> > > > >> >> >
> > > > >> "expected_recovered_size": "0b",
> > > > >> >> >
> > > > >> "expected_recovered_size_in_bytes": 0,
> > > > >> >> >
> > > > "recovered_size": "0b",
> > > > >> >> >
> > > > >> "recovered_size_in_bytes": 0,
> > > > >> >> >
> > > "reused_size": "0b",
> > > > >> >> >
> > > > "reused_size_in_bytes": 0,
> > > > >> >> >
> > "size": "0b",
> > > > >> >> >
> > > "size_in_bytes": 0,
> > > > >> >> >
> > > > "throttling_time": "0s",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 0
> > > > >> >> > },
> > > > >> >> >
> > "stage": "RETRY",
> > > > >> >> >
> > > "start_time_in_millis":
> > > > >> 1282226019603,
> > > > >> >> >
> > "throttling_time":
> > > > "7.6h",
> > > > >> >> >
> > > > >> "throttling_time_in_millis": 27514627,
> > > > >> >> >
> > "time": "7.6h",
> > > > >> >> >
> > "time_in_millis":
> > > > 27514657,
> > > > >> >> >
> > "translog": {
> > > > >> >> >
> > "recovered": 0
> > > > >> >> > }
> > > > >> >> > },
> > > > >> >> > "index":
> > 
> > ```
> 
> {
> 
> > ```
> > > > >> >> >
> > "size": "0b",
> > > > >> >> >
> > "size_in_bytes": 0
> > > > >> >> > },
> > > > >> >> >
> > 
> > ```
> 
> "routing": {
> 
> > ```
> > > > >> >> >
> > 
> > ```
> 
> "index":
> 
> > ```
> > > "twitter",
> > > > >> >> >
> > 
> > ```
> 
> "node":
> 
> > ```
> > > > >> >>
> > "031642a1-968f-40fb-b7c2-5a869769d5b4",
> > > > >> >> >
> > "primary": true,
> > > > >> >> >
> > > "relocating_node": null,
> > > > >> >> >
> > "shard": 1,
> > > > >> >> >
> > 
> > ```
> 
> "state":
> 
> > ```
> > > "INITIALIZING"
> > > > >> >> > },
> > > > >> >> > "state":
> > "RECOVERING"
> > > > >> >> > }
> > > > >> >> > ]
> > > > >> >>
> > > > >> >>
> > > > >> >> Shay Banon wrote:
> > > > >> >> > It should be allocated on the
> > other node, you
> > > > shouldn't
> > > > >> need to start
> > > > >> >> > another node. When you issue a
> > cluster health
> > > > (simple
> > > > >> curl can
> > > > >> >> do), what
> > > > >> >> > is the status? The cluster state
> > API gives
> > > you more
> > > > >> information if you
> > > > >> >> > are after (each shard and its
> > state).
> > > > >> >> >
> > > > >> >> > On Thu, Aug 19, 2010 at 3:48 PM,
> > Kenneth
> > > Loafman
> > > > >> >> > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>>> wrote:
> > > > >> >> >
> > > > >> >> > No this is the first time.
> > 
> > ```
> 
> The
> 
> > ```
> > > shutdown took a
> > > > >> while with several
> > > > >> >> > 'Waiting for not to
> > shutdown..." style
> > > > message. It
> > > > >> came up bad
> > > > >> >> > after that.
> > > > >> >> >
> > > > >> >> > So, if I have two nodes now,
> > and one
> > > needs to be
> > > > >> recovered,
> > > > >> >> I'll need 3
> > > > >> >> > nodes to get the recovery
> > 
> > ```
> 
> done?
> 
> > ```
> > > > >> >> >
> > > > >> >> > ...Ken
> > > > >> >> >
> > > > >> >> > Shay Banon wrote:
> > > > >> >> > > The shard will allocated
> > to another
> > > node and
> > > > >> recovered there. Do
> > > > >> >> > you see
> > > > >> >> > > it happen continuously?
> > > > >> >> > >
> > > > >> >> > > -shay.banon
> > > > >> >> > >
> > > > >> >> > > On Thu, Aug 19, 2010 at
> > 2:28 PM, Kenneth
> > > > Loafman
> > > > >> >> > > <kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>>
> > > > >> >> >
> > <mailto:kenneth.loafman@gmail.com <mailto:
> > 
> > ```
> 
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\>
> 
> > ```
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>
> > > > >> >> >
> > <mailto:kenneth.loafman@gmail.com <mailto:
> > 
> > ```
> 
> [kenneth.loafman@gmail.com](mailto:kenneth.loafman@gmail.com)\>
> 
> > ```
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>
> > > > >> >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>
> > > > >> <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>
> > > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>
> > > <mailto:kenneth.loafman@gmail.com
> > <mailto:kenneth.loafman@gmail.com>>>>>>>> wrote:
> > > > >> >> > >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > > The second shard on
> > one of my
> > > indexes has
> > > > >> failed due to:
> > > > >> >> > > [05:59:47,332][WARN
> > ][index.gateway
> > > > >> ] [Mangog]
> > > > >> >> > [twitter][1]
> > > > >> >> > > failed to snapshot on
> > close
> > > > >> >> > > ...followed by a long
> > traceback.
> > > > >> >> > > ...followed by:
> > > > >> >> > > [05:59:49,336][WARN
> > > ][cluster.action.shard
> > > > >> ] [Mangog]
> > > > >> >> > received shard
> > > > >> >> > > failed for
> > 
> > ```
> 
> [twitter][1],
> 
> > ```
> > > > >> >> >
> > > node[86d601df-e124-45ed-a5f2-57d762042d87],
> > > > >> >> > > [P], s[INITIALIZING],
> > reason [Failed
> > > > to start
> > > > >> shard, message
> > > > >> >> > >
> > > > >>
> > [IndexShardGatewayRecoveryException[[twitter][1]
> > > Failed to
> > > > >> >> > recovery
> > > > >> >> > > translog]; nested:
> > > > >> >>
> > EngineCreationFailureException[[twitter][1]
> > > > >> >> > Failed to
> > > > >> >> > > open reader on
> > writer]; nested:
> > > > >> >> > >
> > > > >> >> >
> > > > >> >>
> > > > >>
> > > >
> > >
> > 
> > ```
> 
> FileNotFoundException[/mnt/search-data-dev/elasticsearch/nodes/0/indices/twitter/1/index/\_d8g.cfs
> 
> > ```
> > > > >> >> > > (No such file or
> > directory)]; ]]
> > > > >> >> > >
> > > > >> >> > > Is the recovery process
> > > automatic, or do I
> > > > >> have to do
> > > > >> >> something
> > > > >> >> > > special? It appears
> > to be just this
> > > > one shard.
> > > > >> >> > >
> > > > >> >> > > I use the service
> > wrapper to
> > > start/stop
> > > > >> 0.9.1-SNAPSHOT,
> > > > >> >> and my
> > > > >> >> > config is
> > > > >> >> > > below.
> > > > >> >> > >
> > > > >> >> > > ...Thanks,
> > > > >> >> > > ...Ken
> > > > >> >> > >
> > > > >> >> > > cloud:
> > > > >> >> > > aws:
> > > > >> >> > > access_key:
> > 
> > ```
> 
> * * *
> 
> > ```
> > > > >> >> > > secret_key:
> > 
> > ```
> 
> * * *
> 
> > ```
> > > > >> >> > >
> > > > >> >> > > gateway:
> > > > >> >> > > type: s3
> > > > >> >> > > s3:
> > > > >> >> > > bucket: *****
> > > > >> >> > >
> > > > >> >> > > path :
> > > > >> >> > > work :
> > /mnt/search-data-dev
> > > > >> >> > > logs :
> > > /mnt/search-data-dev/node1/logs
> > > > >> >> > >
> > > > >> >> > > index :
> > > > >> >> > > number_of_shards : 2
> > > > >> >> > > number_of_replicas :
> > 
> > ```
> 
> 1
> 
> > ```
> > > > >> >> > >
> > > > >> >> > > network :
> > > > >> >> > > host : 192.168.1.5
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> >
> > > > >> >> >
> > > > >> >>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >
> > > >
> > > >
> > >
> > >
> > 
> > ```

[Next page](https://discuss.elastic.co/t/how-to-restart-recover-a-shard/3239.md?page=2)