# ES Cluster in Yellow with Unassigned Shards flopping in and out of Unassigned
**URL:** https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725
**Category:** Elasticsearch
**Created:** [December 22, 2016, 12:57am UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725 "2016-12-22T00:57:37Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@jquinnett](https://discuss.elastic.co/u/jquinnett)
#### Post date: [December 22, 2016, 12:57am UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725/1 "2016-12-22T00:57:37Z")
im having a problem with my ES cluster. When i check cluster status im seeing Yellow... I see 1 min unassigned Shards = 4 and next min its 0
curl -XGET '[http://elasticsearch:9200/\_cluster/health?pretty](http://elasticsearch:9200/_cluster/health?pretty)'
{
"cluster\_name" : "ES\_cluster",
"status" : "yellow",
"timed\_out" : false,
"number\_of\_nodes" : 8,
"number\_of\_data\_nodes" : 5,
"active\_primary\_shards" : 106,
"active\_shards" : 208,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 4,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 6,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 31,
"active\_shards\_percent\_as\_number" : 98.11320754716981
}
curl -XGET '[http://elasticsearch:9200/\_cluster/health?pretty](http://elasticsearch:9200/_cluster/health?pretty)'
{
"cluster\_name" : "ES\_cluster",
"status" : "yellow",
"timed\_out" : false,
"number\_of\_nodes" : 8,
"number\_of\_data\_nodes" : 5,
"active\_primary\_shards" : 106,
"active\_shards" : 208,
"relocating\_shards" : 0,
"initializing\_shards" : 4,
"unassigned\_shards" : 0,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 5,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 118,
"active\_shards\_percent\_as\_number" : 98.11320754716981
}
Its flopping back and forth between initializing shards and unassigned shards. Earlier in the day another engineer deleted some large old indexes, which might have caused this problem.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 22, 2016, 2:24am UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725/2 "2016-12-22T02:24:29Z")
What do the logs on the master node show?
> [@jquinnett](#):
>
> Earlier in the day another engineer deleted some large old indexes, which might have caused this problem
Unlikely.
---
### Author:  [@jquinnett](https://discuss.elastic.co/u/jquinnett)
#### Post date: [December 22, 2016, 4:04pm UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725/3 "2016-12-22T16:04:07Z")
Lots of Warnings about not being able to receive shard due to index that other engineer delete being missing.
[2016-12-21 23:45:02,100][WARN][cluster.action.shard] [elasticsearch-master0.node.netprod.vci] [logstash-2016.12.02][2] received shard failed for [logstash-2016.12.02][2], node[u7ReG8ycQ1udVr8j-5-nyA], [R], v[102954], s[INITIALIZING], a[id=bLbgpONHQKe5YDyGa3Qmmw], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2016-12-21T23:45:01.980Z], details[failed recovery, failure RecoveryFailedException[[logstash-2016.12.02][2]: Recovery failed from {elasticsearch-data1.node.netprod.vci}{6EfoOppSQ\_q-ANHPNP2H0g}{192.99.99.59}{192.99.99.59:9300}{master=false} into {elasticsearch-data0.node.netprod.vci}{mVot4nbASjqU4Ij1eGAhwA}{192.99.99.44}{192.99.99.44:9300}{master=false}]; nested: RemoteTransportException[[elasticsearch-data1.node.netprod.vci][192.99.99.59:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index]; ]], indexUUID [OVg7952VR8uenasSv-16iw], message [failed recovery], failure [RecoveryFailedException[[logstash-2016.12.02][2]: Recovery failed from {elasticsearch-data1.node.netprod.vci}{6EfoOppSQ\_q-ANHPNP2H0g}{192.99.99.59}{192.99.99.59:9300}{master=false} into {elasticsearch-data4.node.netprod.vci}{u7ReG8ycQ1udVr8j-5-nyA}{192.99.99.50}{192.99.99.50:9300}{master=false}]; nested: RemoteTransportException[[elasticsearch-data1.node.netprod.vci][192.99.99.59:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index]; ]
RecoveryFailedException[[logstash-2016.12.02][2]: Recovery failed from {elasticsearch-data1.node.netprod.vci}{6EfoOppSQ\_q-ANHPNP2H0g}{192.99.99.59}{192.99.99.59:9300}{master=false} into {elasticsearch-data4.node.netprod.vci}{u7ReG8ycQ1udVr8j-5-nyA}{192.99.99.50}{192.99.99.50:9300}{master=false}]; nested: RemoteTransportException[[elasticsearch-data1.node.netprod.vci][192.99.99.59:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index];
at org.elasticsearch.indices.recovery.RecoveryTarget.doRecovery(RecoveryTarget.java:258)
at org.elasticsearch.indices.recovery.RecoveryTarget.access$1100(RecoveryTarget.java:69)
at org.elasticsearch.indices.recovery.RecoveryTarget$RecoveryRunner.doRun(RecoveryTarget.java:508)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: RemoteTransportException[[elasticsearch-data1.node.netprod.vci][192.99.99.59:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index];
Caused by: [logstash-2016.12.02][[logstash-2016.12.02][2]] RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index];
at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
at org.elasticsearch.indices.recovery.RecoverySourceHandler.recoverToTarget(RecoverySourceHandler.java:135)
at org.elasticsearch.indices.recovery.RecoverySource.recover(RecoverySource.java:126)
at org.elasticsearch.indices.recovery.RecoverySource.access$200(RecoverySource.java:52)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:135)
at org.elasticsearch.indices.recovery.RecoverySource$StartRecoveryTransportRequestHandler.messageReceived(RecoverySource.java:132)
at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:299)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: [logstash-2016.12.02][[logstash-2016.12.02][2]] RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: NoSuchFileException[/var/log/elasticsearch/netprod/nodes/0/indices/logstash-2016.12.02/2/index];
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 22, 2016, 9:15pm UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725/4 "2016-12-22T21:15:34Z")
> [@jquinnett](#):
>
> Lots of Warnings about not being able to receive shard due to index that other engineer delete being missing.
How were the indices deleted?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 19, 2017, 9:15pm UTC](https://discuss.elastic.co/t/es-cluster-in-yellow-with-unassigned-shards-flopping-in-and-out-of-unassigned/69725/5 "2017-01-19T21:15:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.