# Index recovery failure on node restart since v1.3.x
**URL:** https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126
**Category:** Elasticsearch
**Created:** [October 7, 2014, 8:56pm UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126 "2014-10-07T20:56:31Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)
#### Post date: [October 7, 2014, 8:56pm UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/1 "2014-10-07T20:56:31Z")
We have a single node ES instance, which is restarted once a week. Every
time it's restarted, one specific index recovery is always stuck at -
> [2014-10-06 22:47:48,107][DEBUG][index.translog] [testnode]
> [testindex\_20140930][0] interval [5s], flush\_threshold\_ops [2147483647],
> flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> [2014-10-06 22:47:48,108][DEBUG][index.shard.service] [testnode]
> [testindex\_20140930][0] state: [CREATED]-\>[RECOVERING], reason [from
> gateway]
> [2014-10-06 22:47:48,108][DEBUG][index.gateway] [testnode]
> [testindex\_20140930][0] starting recovery from local ...
> [2014-10-06 22:47:48,203][DEBUG][index.engine.internal] [testnode]
> [testindex\_20140930][0] starting engine
>
> We have to delete that index for recovery to complete. Doing hot threads
> dump, we get following logs -
> :::
> [testnode.node][ff9m9KnRSqWfkrTZiAMbsA][testnode][inet[/10.126.143.197:9301]]{datacenter=nj,
> master=true}
102.9% (514.3ms out of 500ms) cpu usage by thread
'elasticsearch[testnode.node][generic][T#2]'
10/10 snapshots sharing following 14 elements
org.elasticsearch.index.engine.internal.InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1574)
org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:160)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:122)
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:779)
org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:686)
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:780)
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:250)
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
java.lang.Thread.run(Thread.java:722)
We started seeing this error with upgrade to v1.3.2, and still happening
with v1.3.4. Could someone advice what could be happening? Thanks.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 7, 2014, 9:00pm UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/2 "2014-10-07T21:00:46Z")
Why are you restarting the node every week?
That sounds like a problem you should solve to stop this one happening.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 8 October 2014 07:56, Ankush Jhalani [ankush.jhalani@gmail.com](mailto:ankush.jhalani@gmail.com) wrote:
> We have a single node ES instance, which is restarted once a week. Every
> time it's restarted, one specific index recovery is always stuck at -
>
> > [2014-10-06 22:47:48,107][DEBUG][index.translog] [testnode]
> > [testindex\_20140930][0] interval [5s], flush\_threshold\_ops [2147483647],
> > flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> > [2014-10-06 22:47:48,108][DEBUG][index.shard.service] [testnode]
> > [testindex\_20140930][0] state: [CREATED]-\>[RECOVERING], reason [from
> > gateway]
> > [2014-10-06 22:47:48,108][DEBUG][index.gateway] [testnode]
> > [testindex\_20140930][0] starting recovery from local ...
> > [2014-10-06 22:47:48,203][DEBUG][index.engine.internal] [testnode]
> > [testindex\_20140930][0] starting engine
> >
> > We have to delete that index for recovery to complete. Doing hot threads
> > dump, we get following logs -
> > :::
> > [testnode.node][ff9m9KnRSqWfkrTZiAMbsA][testnode][inet[/10.126.143.197:9301]]{datacenter=nj,
> > master=true}
>
> 102.9% (514.3ms out of 500ms) cpu usage by thread
> 'elasticsearch[testnode.node][generic][T#2]'
> 10/10 snapshots sharing following 14 elements
>
> org.elasticsearch.index.engine.internal.InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1574)
>
> org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:160)
>
> org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:122)
>
> org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
>
> org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
>
> org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
>
> org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:779)
>
> org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:686)
>
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:780)
>
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:250)
>
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> java.lang.Thread.run(Thread.java:722)
>
> We started seeing this error with upgrade to v1.3.2, and still happening
> with v1.3.4. Could someone advice what could be happening? Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bFFFJ3mLmFxn2fYZmb%2BAH0k9aZzJ\_bK6dVoR-sOQZ%2B6Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bFFFJ3mLmFxn2fYZmb%2BAH0k9aZzJ_bK6dVoR-sOQZ%2B6Q%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)
#### Post date: [October 7, 2014, 9:03pm UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/3 "2014-10-07T21:03:29Z")
Well it's a shared resource (not prod), used for other stuff and due to
historical/enterprise reasons it's bounced every week. Though not ideal, I
expect ES to be able to restart without issues.
On Tuesday, October 7, 2014 5:01:15 PM UTC-4, Mark Walkom wrote:
> Why are you restarting the node every week?
> That sounds like a problem you should solve to stop this one happening.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 8 October 2014 07:56, Ankush Jhalani \<[ankush....@gmail.com](mailto:ankush....@gmail.com)
> \\> wrote:
>
> > We have a single node ES instance, which is restarted once a week. Every
> > time it's restarted, one specific index recovery is always stuck at -
> >
> > > [2014-10-06 22:47:48,107][DEBUG][index.translog] [testnode]
> > > [testindex\_20140930][0] interval [5s], flush\_threshold\_ops [2147483647],
> > > flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> > > [2014-10-06 22:47:48,108][DEBUG][index.shard.service] [testnode]
> > > [testindex\_20140930][0] state: [CREATED]-\>[RECOVERING], reason [from
> > > gateway]
> > > [2014-10-06 22:47:48,108][DEBUG][index.gateway] [testnode]
> > > [testindex\_20140930][0] starting recovery from local ...
> > > [2014-10-06 22:47:48,203][DEBUG][index.engine.internal] [testnode]
> > > [testindex\_20140930][0] starting engine
> > >
> > > We have to delete that index for recovery to complete. Doing hot
> > > threads dump, we get following logs -
> > > :::
> > > [testnode.node][ff9m9KnRSqWfkrTZiAMbsA][testnode][inet[/10.126.143.197:9301]]{datacenter=nj,
> > > master=true}
> >
> > 102.9% (514.3ms out of 500ms) cpu usage by thread
> > 'elasticsearch[testnode.node][generic][T#2]'
> > 10/10 snapshots sharing following 14 elements
> >
> > org.elasticsearch.index.engine.internal.InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1574)
> >
> > org.apache.lucene.search.SearcherManager.getSearcher(SearcherManager.java:160)
> >
> > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:122)
> >
> > org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58)
> >
> > org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176)
> >
> > org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:225)
> >
> > org.elasticsearch.index.engine.internal.InternalEngine.refresh(InternalEngine.java:779)
> >
> > org.elasticsearch.index.engine.internal.InternalEngine.delete(InternalEngine.java:686)
> >
> > org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryOperation(InternalIndexShard.java:780)
> >
> > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:250)
> >
> > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> >
> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> >
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> > java.lang.Thread.run(Thread.java:722)
> >
> > We started seeing this error with upgrade to v1.3.2, and still happening
> > with v1.3.4. Could someone advice what could be happening? Thanks.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Thibaut\_Britz\_2](https://discuss.elastic.co/u/Thibaut_Britz_2)
#### Post date: [October 8, 2014, 10:04am UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/4 "2014-10-08T10:04:44Z")
Hi,
I would open up an issue on github. Even if it's just one node,
elasticsearch should restart.
Thanks,
Thibaut
On Tue, Oct 7, 2014 at 11:03 PM, Ankush Jhalani [ankush.jhalani@gmail.com](mailto:ankush.jhalani@gmail.com)
wrote:
> Well it's a shared resource (not prod), used for other stuff and due to
> historical/enterprise reasons it's bounced every week. Though not ideal, I
> expect ES to be able to restart without issues.
>
> On Tuesday, October 7, 2014 5:01:15 PM UTC-4, Mark Walkom wrote:
>
> > Why are you restarting the node every week?
> > That sounds like a problem you should solve to stop this one happening.
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 8 October 2014 07:56, Ankush Jhalani [ankush....@gmail.com](mailto:ankush....@gmail.com) wrote:
> >
> > > We have a single node ES instance, which is restarted once a week. Every
> > > time it's restarted, one specific index recovery is always stuck at -
> > >
> > > > [2014-10-06 22:47:48,107][DEBUG][index.translog] [testnode]
> > > > [testindex\_20140930][0] interval [5s], flush\_threshold\_ops [2147483647],
> > > > flush\_threshold\_size [200mb], flush\_threshold\_period [30m]
> > > > [2014-10-06 22:47:48,108][DEBUG][index.shard.service] [testnode]
> > > > [testindex\_20140930][0] state: [CREATED]-\>[RECOVERING], reason [from
> > > > gateway]
> > > > [2014-10-06 22:47:48,108][DEBUG][index.gateway] [testnode]
> > > > [testindex\_20140930][0] starting recovery from local ...
> > > > [2014-10-06 22:47:48,203][DEBUG][index.engine.internal] [testnode]
> > > > [testindex\_20140930][0] starting engine
> > > >
> > > > We have to delete that index for recovery to complete. Doing hot
> > > > threads dump, we get following logs -
> > > > ::: [testnode.node][ff9m9KnRSqWfkrTZiAMbsA][testnode][inet[/10.126.143.197:9301]]{datacenter=nj,
> > > > master=true}
> > >
> > > 102.9% (514.3ms out of 500ms) cpu usage by thread
> > > 'elasticsearch[testnode.node][generic][T#2]'
> > > 10/10 snapshots sharing following 14 elements
> > > org.elasticsearch.index.engine.internal.
> > > InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1574)
> > > org.apache.lucene.search.SearcherManager.getSearcher(
> > > SearcherManager.java:160)
> > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(
> > > SearcherManager.java:122)
> > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(
> > > SearcherManager.java:58)
> > > org.apache.lucene.search.ReferenceManager.doMaybeRefresh(
> > > ReferenceManager.java:176)
> > > org.apache.lucene.search.ReferenceManager.maybeRefresh(
> > > ReferenceManager.java:225)
> > > org.elasticsearch.index.engine.internal.InternalEngine.refresh(
> > > InternalEngine.java:779)
> > > org.elasticsearch.index.engine.internal.InternalEngine.delete(
> > > InternalEngine.java:686)
> > > org.elasticsearch.index.shard.service.InternalIndexShard.
> > > performRecoveryOperation(InternalIndexShard.java:780)
> > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.
> > > recover(LocalIndexShardGateway.java:250)
> > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.
> > > run(IndexShardGatewayService.java:132)
> > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > ThreadPoolExecutor.java:1110)
> > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > ThreadPoolExecutor.java:603)
> > > java.lang.Thread.run(Thread.java:722)
> > >
> > > We started seeing this error with upgrade to v1.3.2, and still happening
> > > with v1.3.4. Could someone advice what could be happening? Thanks.
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%
> > > [40googlegroups.com](http://40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAE\_AicSdFYAAh7JXQvgexQwq8rqwzegpQbW6AwRSX0v%2BNDYHGQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE_AicSdFYAAh7JXQvgexQwq8rqwzegpQbW6AwRSX0v%2BNDYHGQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ankush\_Jhalani](https://discuss.elastic.co/u/Ankush_Jhalani)
#### Post date: [October 8, 2014, 3:07pm UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/5 "2014-10-08T15:07:15Z")
Thanks. It's difficult to replicate w/o the data but I will try to ask on
github.
On Wednesday, October 8, 2014 6:04:52 AM UTC-4, Thibaut wrote:
> Hi,
>
> I would open up an issue on github. Even if it's just one node,
> elasticsearch should restart.
>
> Thanks,
> Thibaut
>
> On Tue, Oct 7, 2014 at 11:03 PM, Ankush Jhalani \<[ankush....@gmail.com](mailto:ankush....@gmail.com)
> \\> wrote:
>
> > Well it's a shared resource (not prod), used for other stuff and due to
> > historical/enterprise reasons it's bounced every week. Though not ideal, I
> > expect ES to be able to restart without issues.
> >
> > On Tuesday, October 7, 2014 5:01:15 PM UTC-4, Mark Walkom wrote:
> >
> > > Why are you restarting the node every week?
> > > That sounds like a problem you should solve to stop this one happening.
> > >
> > > Regards,
> > > Mark Walkom
> > >
> > > Infrastructure Engineer
> > > Campaign Monitor
> > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> > >
> > > On 8 October 2014 07:56, Ankush Jhalani [ankush....@gmail.com](mailto:ankush....@gmail.com) wrote:
> > >
> > > > We have a single node ES instance, which is restarted once a week.
> > > > Every time it's restarted, one specific index recovery is always stuck at -
> > > >
> > > > > [2014-10-06 22:47:48,107][DEBUG][index.translog]
> > > > > [testnode] [testindex\_20140930][0] interval [5s], flush\_threshold\_ops
> > > > > [2147483647], flush\_threshold\_size [200mb], flush\_threshold\_period
> > > > > [30m]
> > > > > [2014-10-06 22:47:48,108][DEBUG][index.shard.service]
> > > > > [testnode] [testindex\_20140930][0] state: [CREATED]-\>[RECOVERING], reason
> > > > > [from gateway]
> > > > > [2014-10-06 22:47:48,108][DEBUG][index.gateway]
> > > > > [testnode] [testindex\_20140930][0] starting recovery from local ...
> > > > > [2014-10-06 22:47:48,203][DEBUG][index.engine.internal]
> > > > > [testnode] [testindex\_20140930][0] starting engine
> > > > >
> > > > > We have to delete that index for recovery to complete. Doing hot
> > > > > threads dump, we get following logs -
> > > > > ::: [testnode.node][ff9m9KnRSqWfkrTZiAMbsA][testnode][inet[/10.126.143.197:9301]]{datacenter=nj,
> > > > > master=true}
> > > >
> > > > 102.9% (514.3ms out of 500ms) cpu usage by thread
> > > > 'elasticsearch[testnode.node][generic][T#2]'
> > > > 10/10 snapshots sharing following 14 elements
> > > > org.elasticsearch.index.engine.internal.
> > > > InternalEngine$SearchFactory.newSearcher(InternalEngine.java:1574)
> > > > org.apache.lucene.search.SearcherManager.getSearcher(
> > > > SearcherManager.java:160)
> > > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(
> > > > SearcherManager.java:122)
> > > > org.apache.lucene.search.SearcherManager.refreshIfNeeded(
> > > > SearcherManager.java:58)
> > > > org.apache.lucene.search.ReferenceManager.doMaybeRefresh(
> > > > ReferenceManager.java:176)
> > > > org.apache.lucene.search.ReferenceManager.maybeRefresh(
> > > > ReferenceManager.java:225)
> > > > org.elasticsearch.index.engine.internal.InternalEngine.refresh(
> > > > InternalEngine.java:779)
> > > > org.elasticsearch.index.engine.internal.InternalEngine.delete(
> > > > InternalEngine.java:686)
> > > > org.elasticsearch.index.shard.service.InternalIndexShard.
> > > > performRecoveryOperation(InternalIndexShard.java:780)
> > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.
> > > > recover(LocalIndexShardGateway.java:250)
> > > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.
> > > > run(IndexShardGatewayService.java:132)
> > > > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > > > ThreadPoolExecutor.java:1110)
> > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > > > ThreadPoolExecutor.java:603)
> > > > java.lang.Thread.run(Thread.java:722)
> > > >
> > > > We started seeing this error with upgrade to v1.3.2, and still
> > > > happening with v1.3.4. Could someone advice what could be happening? Thanks.
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%
> > > > [40googlegroups.com](http://40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/584e7b07-0957-49ca-b67a-3f8dc281312a%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/327c2b19-109a-4f42-9031-93a2c8c275e9%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e68dff51-c65f-4149-b693-048011326a73%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e68dff51-c65f-4149-b693-048011326a73%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:57am UTC](https://discuss.elastic.co/t/index-recovery-failure-on-node-restart-since-v1-3-x/20126/6 "2017-07-06T00:57:36Z")