# Lost shards and cluster state stays red

**URL:** https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746
**Category:** Elasticsearch
**Created:** [November 16, 2012, 11:23pm UTC](https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746 "2012-11-16T23:23:09Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)
#### Post date: [November 16, 2012, 11:23pm UTC](https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746/1 "2012-11-16T23:23:09Z")

</div>

I had a hardware trouble during ES was in yellow state.  
Some shards that were in the troubled disk were gone.  
It seems I lost some primary shards because it was in yellow state and some shards that were in the troubled node did not have its replica.  
(my number\_of\_replicas is 1)

The shard reallocation has finished but the cluster state stays red.  
{"cluster\_name":"es-cluster1",  
"status":"red",  
"timed\_out":false,  
"number\_of\_nodes":16,  
"number\_of\_data\_nodes":16,  
"active\_primary\_shards":2469,  
"active\_shards":4938,  
"relocating\_shards":0,  
"initializing\_shards":0,  
"unassigned\_shards":118}

It should have 2528 primary shards but (2528 - 2469 = 59) shards are missing.

I'll reindex documents belonged to the lost shards, but is there any way to put the cluster back to yellow/green state, so that at the same time I can update index and search existing documents?

Thanks for your help.  
I want to

---

<div class="post-metadata">

### Author: ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [November 17, 2012, 2:08pm UTC](https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746/2 "2012-11-17T14:08:09Z")

</div>

Hello,

I'm not sure if I understood you question correctly, but what you can do is:

- delete the indices which have missing shards. Something like:

# curl -XDELETE localhost:9200/corrupted\_index/

- reindex data belonging to those "incomplete" indices

Then your cluster state should be back to yellow/green again. Until  
then, if you have indices that have missing shards but also allocated  
shards, ES will still run your searches on the data you have. If  
that's important to you, then you might prefer to do things like this:

- reindex data belonging to incomplete indices into new indices with  
different names
- delete indices with missing shards
- add aliases[0] to the new indices with the old index names, so that  
searches will run as before

[0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-aliases.html)

## Best regards, Radu

[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene

On Sat, Nov 17, 2012 at 1:23 AM, arta [artasano@sbcglobal.net](mailto:artasano@sbcglobal.net) wrote:

> I had a hardware trouble during ES was in yellow state.  
> Some shards that were in the troubled disk were gone.  
> It seems I lost some primary shards because it was in yellow state and some  
> shards that were in the troubled node did not have its replica.  
> (my number\_of\_replicas is 1)
> 
> The shard reallocation has finished but the cluster state stays red.  
> {"cluster\_name":"es-cluster1",  
> "status":"red",  
> "timed\_out":false,  
> "number\_of\_nodes":16,  
> "number\_of\_data\_nodes":16,  
> "active\_primary\_shards":2469,  
> "active\_shards":4938,  
> "relocating\_shards":0,  
> "initializing\_shards":0,  
> "unassigned\_shards":118}
> 
> It should have 2528 primary shards but (2528 - 2469 = 59) shards are  
> missing.
> 
> I'll reindex documents belonged to the lost shards, but is there any way to  
> put the cluster back to yellow/green state, so that at the same time I can  
> update index and search existing documents?
> 
> Thanks for your help.  
> I want to
> 
> --  
> View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Lost-shards-and-cluster-state-stays-red-tp4025616.html](http://elasticsearch-users.115913.n3.nabble.com/Lost-shards-and-cluster-state-stays-red-tp4025616.html)  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).
> 
> --

--

---

<div class="post-metadata">

### Author: ![arta](https://avatars.discourse-cdn.com/v4/letter/a/aca169/32.png) [@arta](https://discuss.elastic.co/u/arta)
#### Post date: [November 17, 2012, 7:53pm UTC](https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746/3 "2012-11-17T19:53:26Z")

</div>

Thank you Radu,  
I don't want to delete indices, as they have millions of docs.

I found this topic:

[http://elasticsearch-users.115913.n3.nabble.com/Recovering-after-shard-failure-td4018776.html](http://elasticsearch-users.115913.n3.nabble.com/Recovering-after-shard-failure-td4018776.html)

I created a dummy index, manually copied one of it's shards to the node where shards were lost, renamed shard directory names as they were, started elastic search at the node, and deleted the dummy index.  
ES got back to green.  
Docs in the indices remained (of course except for ones in the lost shards).

--

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:03am UTC](https://discuss.elastic.co/t/lost-shards-and-cluster-state-stays-red/9746/4 "2017-07-06T03:03:50Z")

</div>


