# Use logstash to reindex in the same cluster, but reponse code 503

**URL:** https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939
**Category:** Elasticsearch
**Created:** [March 21, 2016, 1:50am UTC](https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939 "2016-03-21T01:50:26Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![lizhenmxcz](https://avatars.discourse-cdn.com/v4/letter/l/a88e57/32.png) [@lizhenmxcz](https://discuss.elastic.co/u/lizhenmxcz)
#### Post date: [March 21, 2016, 1:50am UTC](https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939/1 "2016-03-21T01:50:26Z")

</div>

I use logstash to reindex in the same cluster, but when it ran more that 24 hours, a error occurred in the logstash with "retrying failed action with response code: 503 {:level=\>:warn}". I use es2.1 and logstash2.1.

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [March 22, 2016, 8:53pm UTC](https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939/2 "2016-03-22T20:53:33Z")

</div>

I suspect the ES log contains additional clues. Perhaps you don't have sufficient cluster capacity to reindex all the data?

---

<div class="post-metadata">

### Author: ![lizhenmxcz](https://avatars.discourse-cdn.com/v4/letter/l/a88e57/32.png) [@lizhenmxcz](https://discuss.elastic.co/u/lizhenmxcz)
#### Post date: [March 23, 2016, 1:50am UTC](https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939/3 "2016-03-23T01:50:51Z")

</div>

this is the log.  
[2016-03-20 23:59:56,746][DEBUG][action.admin.indices.stats] [bigdata10] [indices:monitor/stats] failed to execute operation for shard [[armin2015\_0507][0], node[Ky-5cgAVSoGiMlEfso235w], [P], v[5], s[INITIALIZING], a[id=3-z6d-jQS1KS03n7GB097A], unassigned\_info[[reason=NODE\_LEFT], at[2016-03-19T23:49:28.145Z], details[node\_left[Ky-5cgAVSoGiMlEfso235w]]]]  
[armin2015\_0507][[armin2015\_0507][0]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST\_RECOVERY, STARTED, RELOCATED]];  
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:405)  
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:382)  
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:371)  
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:299)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: [armin2015\_0507][[armin2015\_0507][0]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST\_RECOVERY, STARTED, RELOCATED]]  
at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:974)  
at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:808)  
at org.elasticsearch.index.shard.IndexShard.docStats(IndexShard.java:628)  
at org.elasticsearch.action.admin.indices.stats.CommonStats.(CommonStats.java:131)  
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:165)  
at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.shardOperation(TransportIndicesStatsAction.java:47)  
at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:401)  
... 7 more

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:06pm UTC](https://discuss.elastic.co/t/use-logstash-to-reindex-in-the-same-cluster-but-reponse-code-503/44939/4 "2017-07-05T23:06:02Z")

</div>


