# Recovering elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [November 24, 2015, 10:04pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496 "2015-11-24T22:04:57Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 24, 2015, 10:04pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/1 "2015-11-24T22:04:57Z")
I am getting below exception for my elasticsearch cluster. Due to this exceptions it is blocking all the incoming traffic and logs are not seeing in Kibana. [2015-11-24 15:36:34,132][INFO][cluster.metadata] [[node1.domain.com](http://node1.domain.com)] [logstash-2015.11.23] creating index, cause [auto(bulk api)], templates [logstash], shards [8]/[1], mappings [_default_, httpd\_search] [2015-11-24 15:36:35,053][DEBUG][action.admin.indices.stats] [[node1.domain.com](http://node1.domain.com)] [indices:monitor/stats] failed to execute operation for shard [[logstash-2015.11.23][5], node[stz9f0JdTp2YNstANI83WQ], [R], v[3], s[INITIALIZING], a[id=lZ7ECXZ0SiyNT44yAfv-jQ], unassigned\_info[[reason=INDEX\_CREATED], at[2015-11-24T21:36:34.133Z]]] [logstash-2015.11.23][[logstash-2015.11.23][5]] BroadcastShardOperationFailedException[operation indices:monitor/stats failed]; nested: IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST\_RECOVERY, STARTED, RELOCATED]]; at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.onShardOperation(TransportBroadcastByNodeAction.java:399) at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:376) at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$BroadcastByNodeTransportRequestHandler.messageReceived(TransportBroadcastByNodeAction.java:365) at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:350) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: [logstash-2015.11.23][[logstash-2015.11.23][5]] IllegalIndexShardStateException[CurrentState[RECOVERING] operations only allowed when shard state is one of [POST\_RECOVERY, STARTED, RELOCATED]] curl -XGET [http://node1.domain.com:9200/\_cluster/health?pretty](http://node1.domain.com:9200/_cluster/health?pretty) { "cluster\_name" : "elasticsearch-npe", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 19, "active\_shards" : 38, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 100.0 } gateway.expected\_nodes: 1 gateway.expected\_master\_nodes: 1 gateway.recover\_after\_nodes: 1 gateway.recover\_after\_master\_nodes: 1 gateway.recover\_after\_data\_nodes: 1 discovery.zen.ping.unicast.hosts: ["[node1.domain.com](http://node1.domain.com)", "[node2.domain.com](http://node2.domain.com)"] node.max\_local\_storage\_nodes: 1 Raghu ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 25, 2015, 1:06am UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/2 "2015-11-25T01:06:27Z")
What version? ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 25, 2015, 2:52am UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/3 "2015-11-25T02:52:13Z")
I am using 2.0 version ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 25, 2015, 8:15pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/4 "2015-11-25T20:15:26Z")
Is this issue because of posted here [Marvel license issue](http://discuss.elastic.co/t/marvel-license-issue/35606) ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 27, 2015, 8:51pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/5 "2015-11-27T20:51:04Z")
Even after upgrading ElasticSearch and Logstash to verions 2.1 and Kibana to 4.3, i am still getting these exceptions. In logstash i could see below error message. "retrying failed action with response code: 503". Even after following steps issue is not getting resolved and log events are not getting indexed. 1. I have disabled the marvel agent. 2. Deleted .kibana index to create new indexes. 3. restarted all the services. Please help me out in resolving this issue. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 27, 2015, 10:38pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/6 "2015-11-27T22:38:21Z")
What's the output from `_cat/shards` look like? ---
**Author:** ![raghu\_ae](https://avatars.discourse-cdn.com/v4/letter/r/a88e4f/32.png) [@raghu\_ae](https://discuss.elastic.co/u/raghu_ae)\ **Post date:** [November 30, 2015, 5:58pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/7 "2015-11-30T17:58:40Z")
All the Indexes shards status is shown as "STARTED" ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:34pm UTC](https://discuss.elastic.co/t/recovering-elasticsearch-cluster/35496/8 "2017-07-05T23:34:48Z")