# Received shard failed on twitter\_river

**URL:** <https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194>\
**Category:** Elasticsearch\
**Created:** [December 20, 2011, 12:36am UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194 "2011-12-20T00:36:49Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\
**Post date:** [December 20, 2011, 12:36am UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/1 "2011-12-20T00:36:49Z")

</div>

Hello,  
We are using ElasticSearch 0.17.4 and had twitter\_river running for  
past 6 months which resulted in OOME error.  
I ended up deleting the twitter plugin folder for jars and relevant  
river\_twitter index  
/data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river

However, even after doing the above and restarting the cluster, I get  
the following in logs:

[2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
the Wisp] sending failed shard for [twitter\_river][3],  
node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
start shard, message  
[IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
for local recovery (post api), should exists, but doesn't]]]  
[2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
the Wisp] received shard failed for [twitter\_river][3],  
node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
start shard, message  
[IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
for local recovery (post api), should exists, but doesn't]]]  
[2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
the Wisp] [twitter\_river][3] failed to start shard  
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
[twitter\_river][3] shard allocated for local recovery (post api),  
should exists, but doesn't  
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)  
[2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
the Wisp] sending failed shard for [twitter\_river][3],  
node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
start shard, message  
[IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
for local recovery (post api), should exists, but doesn't]]]  
[2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
the Wisp] received shard failed for [twitter\_river][3],  
node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
start shard, message  
[IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
for local recovery (post api), should exists, but doesn't]]]  
[2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
the Wisp] [twitter\_river][3] failed to start shard  
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
[twitter\_river][3] shard allocated for local recovery (post api),  
should exists, but doesn't  
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
at java.lang.Thread.run(Thread.java:662)

any idea what might be happening out here?

-anurag

--  
Blog: [http://anurag.co](http://anurag.co)  
Twitter: @anuragphadke ([https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [December 20, 2011, 10:01am UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/2 "2011-12-20T10:01:30Z")

</div>

Deleting the "folders" will not cause the index metadata to be deleted,  
just send a delete index request against that index, it will be removed.

On Tue, Dec 20, 2011 at 2:36 AM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote:

> Hello,  
> We are using Elasticsearch 0.17.4 and had twitter\_river running for  
> past 6 months which resulted in OOME error.  
> I ended up deleting the twitter plugin folder for jars and relevant  
> river\_twitter index  
> /data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river
> 
> However, even after doing the above and restarting the cluster, I get  
> the following in logs:
> 
> [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> the Wisp] sending failed shard for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> the Wisp] received shard failed for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
> the Wisp] [twitter\_river][3] failed to start shard  
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> [twitter\_river][3] shard allocated for local recovery (post api),  
> should exists, but doesn't  
> at  
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> at  
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> the Wisp] sending failed shard for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> the Wisp] received shard failed for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
> the Wisp] [twitter\_river][3] failed to start shard  
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> [twitter\_river][3] shard allocated for local recovery (post api),  
> should exists, but doesn't  
> at  
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> at  
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at  
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)
> 
> any idea what might be happening out here?
> 
> -anurag
> 
> --  
> Blog: [http://anurag.co](http://anurag.co)  
> Twitter: @anuragphadke ([https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\
**Post date:** [December 20, 2011, 1:49pm UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/3 "2011-12-20T13:49:32Z")

</div>

I tried sending it few times, returns back with a {ok} response, but the index doesn't get deleted. Is it because the cluster status is red?

On Dec 20, 2011, at 2:01 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> Deleting the "folders" will not cause the index metadata to be deleted, just send a delete index request against that index, it will be removed.
> 
> On Tue, Dec 20, 2011 at 2:36 AM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote:  
> Hello,  
> We are using Elasticsearch 0.17.4 and had twitter\_river running for  
> past 6 months which resulted in OOME error.  
> I ended up deleting the twitter plugin folder for jars and relevant  
> river\_twitter index  
> /data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river
> 
> However, even after doing the above and restarting the cluster, I get  
> the following in logs:
> 
> [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> the Wisp] sending failed shard for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> the Wisp] received shard failed for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
> the Wisp] [twitter\_river][3] failed to start shard  
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> [twitter\_river][3] shard allocated for local recovery (post api),  
> should exists, but doesn't  
> at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)  
> [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> the Wisp] sending failed shard for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> the Wisp] received shard failed for [twitter\_river][3],  
> node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> start shard, message  
> [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> for local recovery (post api), should exists, but doesn't]]]  
> [2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
> the Wisp] [twitter\_river][3] failed to start shard  
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> [twitter\_river][3] shard allocated for local recovery (post api),  
> should exists, but doesn't  
> at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> at java.lang.Thread.run(Thread.java:662)
> 
> any idea what might be happening out here?
> 
> -anurag
> 
> --  
> Blog: [http://anurag.co](http://anurag.co)  
> Twitter: @anuragphadke ([https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [December 20, 2011, 2:55pm UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/4 "2011-12-20T14:55:42Z")

</div>

No, delete index will be allowed to get in. Have you waited a bit and the  
failures still continue?

On Tue, Dec 20, 2011 at 3:49 PM, Anurag Phadke [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)wrote:

> I tried sending it few times, returns back with a {ok} response, but the  
> index doesn't get deleted. Is it because the cluster status is red?
> 
> On Dec 20, 2011, at 2:01 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> Deleting the "folders" will not cause the index metadata to be deleted,  
> just send a delete index request against that index, it will be removed.
> 
> On Tue, Dec 20, 2011 at 2:36 AM, Anurag \< [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)  
> [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)\> wrote:
> 
> > Hello,  
> > We are using Elasticsearch 0.17.4 and had twitter\_river running for  
> > past 6 months which resulted in OOME error.  
> > I ended up deleting the twitter plugin folder for jars and relevant  
> > river\_twitter index  
> > /data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river
> > 
> > However, even after doing the above and restarting the cluster, I get  
> > the following in logs:
> > 
> > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > the Wisp] sending failed shard for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > the Wisp] received shard failed for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
> > the Wisp] [twitter\_river][3] failed to start shard  
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > [twitter\_river][3] shard allocated for local recovery (post api),  
> > should exists, but doesn't  
> > at  
> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > at  
> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > the Wisp] sending failed shard for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > the Wisp] received shard failed for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
> > the Wisp] [twitter\_river][3] failed to start shard  
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > [twitter\_river][3] shard allocated for local recovery (post api),  
> > should exists, but doesn't  
> > at  
> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > at  
> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at  
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)
> > 
> > any idea what might be happening out here?
> > 
> > -anurag
> > 
> > --  
> > Blog: [http://anurag.co](http://anurag.co)[http://anurag.co](http://anurag.co)  
> > Twitter: @anuragphadke ( [https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke)  
> > [https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![Anurag\_Phadke](https://avatars.discourse-cdn.com/v4/letter/a/5f9b8f/32.png) [@Anurag\_Phadke](https://discuss.elastic.co/u/Anurag_Phadke)\
**Post date:** [December 20, 2011, 3:18pm UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/5 "2011-12-20T15:18:12Z")

</div>

I waited for about 15m. Lemme retry again and wait for a longer time.

On Dec 20, 2011, at 6:55 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> No, delete index will be allowed to get in. Have you waited a bit and the failures still continue?
> 
> On Tue, Dec 20, 2011 at 3:49 PM, Anurag Phadke [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote:  
> I tried sending it few times, returns back with a {ok} response, but the index doesn't get deleted. Is it because the cluster status is red?
> 
> On Dec 20, 2011, at 2:01 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> > Deleting the "folders" will not cause the index metadata to be deleted, just send a delete index request against that index, it will be removed.
> > 
> > On Tue, Dec 20, 2011 at 2:36 AM, Anurag [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com) wrote:  
> > Hello,  
> > We are using Elasticsearch 0.17.4 and had twitter\_river running for  
> > past 6 months which resulted in OOME error.  
> > I ended up deleting the twitter plugin folder for jars and relevant  
> > river\_twitter index  
> > /data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river
> > 
> > However, even after doing the above and restarting the cluster, I get  
> > the following in logs:
> > 
> > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > the Wisp] sending failed shard for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > the Wisp] received shard failed for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
> > the Wisp] [twitter\_river][3] failed to start shard  
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > [twitter\_river][3] shard allocated for local recovery (post api),  
> > should exists, but doesn't  
> > at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)  
> > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > the Wisp] sending failed shard for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > the Wisp] received shard failed for [twitter\_river][3],  
> > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > start shard, message  
> > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > for local recovery (post api), should exists, but doesn't]]]  
> > [2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
> > the Wisp] [twitter\_river][3] failed to start shard  
> > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > [twitter\_river][3] shard allocated for local recovery (post api),  
> > should exists, but doesn't  
> > at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > at java.lang.Thread.run(Thread.java:662)
> > 
> > any idea what might be happening out here?
> > 
> > -anurag
> > 
> > --  
> > Blog: [http://anurag.co](http://anurag.co)  
> > Twitter: @anuragphadke ([https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [December 20, 2011, 3:34pm UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/6 "2011-12-20T15:34:43Z")

</div>

Just make sure you delete the relevant index.

On Tue, Dec 20, 2011 at 5:18 PM, Anurag Phadke [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)wrote:

> I waited for about 15m. Lemme retry again and wait for a longer time.
> 
> On Dec 20, 2011, at 6:55 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:
> 
> No, delete index will be allowed to get in. Have you waited a bit and the  
> failures still continue?
> 
> On Tue, Dec 20, 2011 at 3:49 PM, Anurag Phadke \< [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)  
> [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)\> wrote:
> 
> > I tried sending it few times, returns back with a {ok} response, but the  
> > index doesn't get deleted. Is it because the cluster status is red?
> > 
> > On Dec 20, 2011, at 2:01 AM, Shay Banon \< [kimchy@gmail.com](mailto:kimchy@gmail.com)  
> > [kimchy@gmail.com](mailto:kimchy@gmail.com)\> wrote:
> > 
> > Deleting the "folders" will not cause the index metadata to be deleted,  
> > just send a delete index request against that index, it will be removed.
> > 
> > On Tue, Dec 20, 2011 at 2:36 AM, Anurag \< [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)[anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)  
> > [anurag.phadke@gmail.com](mailto:anurag.phadke@gmail.com)\> wrote:
> > 
> > > Hello,  
> > > We are using Elasticsearch 0.17.4 and had twitter\_river running for  
> > > past 6 months which resulted in OOME error.  
> > > I ended up deleting the twitter plugin folder for jars and relevant  
> > > river\_twitter index  
> > > /data01/elasticsearch/data/buildbot/nodes/\*/indices/twitter\_river
> > > 
> > > However, even after doing the above and restarting the cluster, I get  
> > > the following in logs:
> > > 
> > > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > > the Wisp] sending failed shard for [twitter\_river][3],  
> > > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > > start shard, message  
> > > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > > for local recovery (post api), should exists, but doesn't]]]  
> > > [2011-12-19 16:32:34,086][WARN][cluster.action.shard] [Will o'  
> > > the Wisp] received shard failed for [twitter\_river][3],  
> > > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > > start shard, message  
> > > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > > for local recovery (post api), should exists, but doesn't]]]  
> > > [2011-12-19 16:32:34,115][WARN][indices.cluster] [Will o'  
> > > the Wisp] [twitter\_river][3] failed to start shard  
> > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > > [twitter\_river][3] shard allocated for local recovery (post api),  
> > > should exists, but doesn't  
> > > at  
> > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > > at  
> > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > at java.lang.Thread.run(Thread.java:662)  
> > > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > > the Wisp] sending failed shard for [twitter\_river][3],  
> > > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > > start shard, message  
> > > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > > for local recovery (post api), should exists, but doesn't]]]  
> > > [2011-12-19 16:32:34,117][WARN][cluster.action.shard] [Will o'  
> > > the Wisp] received shard failed for [twitter\_river][3],  
> > > node[KcI\_F3nnSWyhvm\_-McxDTg], [P], s[INITIALIZING], reason [Failed to  
> > > start shard, message  
> > > [IndexShardGatewayRecoveryException[[twitter\_river][3] shard allocated  
> > > for local recovery (post api), should exists, but doesn't]]]  
> > > [2011-12-19 16:32:34,192][WARN][indices.cluster] [Will o'  
> > > the Wisp] [twitter\_river][3] failed to start shard  
> > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:  
> > > [twitter\_river][3] shard allocated for local recovery (post api),  
> > > should exists, but doesn't  
> > > at  
> > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:99)  
> > > at  
> > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:179)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)  
> > > at  
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)  
> > > at java.lang.Thread.run(Thread.java:662)
> > > 
> > > any idea what might be happening out here?
> > > 
> > > -anurag
> > > 
> > > --  
> > > Blog: [http://anurag.co](http://anurag.co) [http://anurag.co](http://anurag.co)[http://anurag.co](http://anurag.co)  
> > > Twitter: @anuragphadke ( [https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke)[https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke)  
> > > [https://twitter.com/#!/anuragphadke](https://twitter.com/#!/anuragphadke))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:44am UTC](https://discuss.elastic.co/t/received-shard-failed-on-twitter-river/6194/7 "2017-07-06T03:44:53Z")

</div>