# Unassigned replica shards, and an unused node
**URL:** https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508
**Category:** Elasticsearch
**Created:** [October 31, 2012, 1:53pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508 "2012-10-31T13:53:23Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [October 31, 2012, 1:53pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/1 "2012-10-31T13:53:23Z")
Hi, I have a production cluster (0.19.8) with a number of replica shards
that are unassigned. All primary shards are accounted for, but some no
longer have redundancy (cluster health is yellow). There is also a node
with no shards whatsoever, but seems to be perfectly fine otherwise, and
removing that then rejoining it does nothing other than show as removed and
added in the master's log. I aslo tried the "reroute" api, but that seems
to be a noop on my version, as it just returns 200 and nothing happens.
Without replicas, I'm reluctant to simply restart other nodes just to see
what happens.
Is using the shutdown api (is it in .19.8?) supposed to force the shards to
other nodes, or does it simply stop the jvm?
Any other tips on how to procede?
Thanks.
--
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [November 1, 2012, 7:45am UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/2 "2012-11-01T07:45:37Z")
Hello James,
On Wed, Oct 31, 2012 at 3:53 PM, James Bardin [j.bardin@gmail.com](mailto:j.bardin@gmail.com) wrote:
> Hi, I have a production cluster (0.19.8) with a number of replica shards
> that are unassigned. All primary shards are accounted for, but some no
> longer have redundancy (cluster health is yellow). There is also a node with
> no shards whatsoever, but seems to be perfectly fine otherwise, and removing
> that then rejoining it does nothing other than show as removed and added in
> the master's log. I aslo tried the "reroute" api, but that seems to be a
> noop on my version, as it just returns 200 and nothing happens.
>
> Without replicas, I'm reluctant to simply restart other nodes just to see
> what happens.
>
> Is using the shutdown api (is it in .19.8?) supposed to force the shards to
> other nodes, or does it simply stop the jvm?
The Shutdown API (available in 0.19.8) doesn't relocate the shards
before stopping the JVM. But Elasticsearch should automatically
redistribute replicas to your other nodes so that everything should be
OK eventually. Take a look here:
[http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html](http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html)
> Any other tips on how to procede?
Any clues in the logs of the empty node? If not, I would turn on
debugging and see it it brings out any new info.
Also, what is the state of your unallocated shards? Are they
initializing or simply "not allocated"?
Do you have any shard allocation settings defined?
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
I'd also try disabling replicas and enabling them again using the
Indices Update Settings API:
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
I know, it sounds a lot like "did you try turning it off and on
again?", but who knows 🙂
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
--
---
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [November 1, 2012, 6:21pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/3 "2012-11-01T18:21:57Z")
On Thu, Nov 1, 2012 at 3:45 AM, Radu Gheorghe
[radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote:
> The Shutdown API (available in 0.19.8) doesn't relocate the shards
> before stopping the JVM. But Elasticsearch should automatically
> redistribute replicas to your other nodes so that everything should be
> OK eventually. Take a look here:
> [http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html](http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html)
Ah, I hadn't seen that with the transient setting before. Wish I had.
> Any clues in the logs of the empty node? If not, I would turn on
> debugging and see it it brings out any new info.
Nothing in the logs that I can see. That node did pick up some shards
after restarting nodes which had redundant data though, so there's not
much else I can investigate there.
> Also, what is the state of your unallocated shards? Are they
> initializing or simply "not allocated"?
Yeah, I have one index now where all replicas are simply "UNASSIGNED".
> Do you have any shard allocation settings defined?
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/allocation.html)
Yes, it there are some routing.allocation settings on the index
missing its replicas, from when we had to push some indexes around a
while back. It's my hunch that this is related, as it's now the only
difference between indexes in the cluster. It now has an include.name
and include.tag settings. There are no more node tags, so that can't
match anything now, and I wonder if it's overriding the include.name.
I really want to just remove these settings now, but I haven't found
any way to do so without building a new index.
> I'd also try disabling replicas and enabling them again using the
> Indices Update Settings API:
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html)
Tried that, but new replicas just show up immediately as UNASSIGNED,
and nothing happens.
> I know, it sounds a lot like "did you try turning it off and on
> again?", but who knows 🙂
May have to resort to that. I could get in a point-release update too.
Thanks,
-james
--
---
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [November 1, 2012, 6:38pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/4 "2012-11-01T18:38:35Z")
The Shutdown API (available in 0.19.8) doesn't relocate the shards
> > before stopping the JVM. But Elasticsearch should automatically
> > redistribute replicas to your other nodes so that everything should be
> > OK eventually. Take a look here:
> > [http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html](http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html)
Excluding all allocations didn't work for the index in question. All
other shard moved accordingly though. Nothing in the logs.
> > I know, it sounds a lot like "did you try turning it off and on
> > again?", but who knows 🙂
>
> May have to resort to that. I could get in a point-release update too.
A rolling restart of the service did nothing for this index. It
totally disappeared during recovery, and then only the primaries came
back online. I'll plan on the 0.19.11 update asap, as I see there a
some code changes around index allocation.
--
---
### Author:  [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)
#### Post date: [November 1, 2012, 8:58pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/5 "2012-11-01T20:58:31Z")
Try setting include.name and include.tag to "\*" for this index
On Thursday, November 1, 2012 2:38:38 PM UTC-4, James Bardin wrote:
> The Shutdown API (available in 0.19.8) doesn't relocate the shards
>
> > > before stopping the JVM. But Elasticsearch should automatically
> > > redistribute replicas to your other nodes so that everything should be
> > > OK eventually. Take a look here:
>
> [http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html](http://elasticsearch-users.115913.n3.nabble.com/how-do-I-relocate-shards-from-a-node-prior-to-shutting-it-down-td4024570.html)
>
> > >
>
> Excluding all allocations didn't work for the index in question. All
> other shard moved accordingly though. Nothing in the logs.
>
> > > I know, it sounds a lot like "did you try turning it off and on
> > > again?", but who knows 🙂
> >
> > May have to resort to that. I could get in a point-release update too.
>
> A rolling restart of the service did nothing for this index. It
> totally disappeared during recovery, and then only the primaries came
> back online. I'll plan on the 0.19.11 update asap, as I see there a
> some code changes around index allocation.
--
---
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [November 1, 2012, 9:09pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/6 "2012-11-01T21:09:36Z")
On Thu, Nov 1, 2012 at 4:58 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:
> Try setting include.name and include.tag to "\*" for this index
Tried that too with no results. I wonder if include.tag takes precedence.
I noticed that @kimchy recently changed the routing allocation code to
treat an empty string as not set -- hoping that alleviates the
problem. It would seem there has to be some way to remove the i.r.a
settings on this index (maybe the java api? I haven't checked it out
at all yet).
Thanks,
-james
--
---
### Author:  [@Shairon\_Toledo](https://discuss.elastic.co/u/Shairon_Toledo)
#### Post date: [November 1, 2012, 9:18pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/7 "2012-11-01T21:18:20Z")
It happened with us some times, the only way that I could to force shards
allocation was by \_id(ids separated by comma), like this
```
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.include._id" :
```
"YxZ92dZCTE2QTUHMrf9s5Q,eBSlwQ2QRNu4hs1mcjWGSQ,XelXrOfRTqmZiP1H\_9X6uw",
"cluster.routing.allocation.cluster\_concurrent\_rebalance": 10
}
}'
Or using by index
curl -XPUT location:9200/index\_name/\_settings -d '{
"index.routing.allocation.include.\_id" : "eBSlwQ2QRNu4hs1mcjWGSQ"
}'
The problem is, in case any node restart I need perform the PUT again.
On Thu, Nov 1, 2012 at 7:09 PM, James Bardin [j.bardin@gmail.com](mailto:j.bardin@gmail.com) wrote:
> On Thu, Nov 1, 2012 at 4:58 PM, Igor Motov [imotov@gmail.com](mailto:imotov@gmail.com) wrote:
>
> > Try setting include.name and include.tag to "\*" for this index
>
> Tried that too with no results. I wonder if include.tag takes precedence.
>
> I noticed that @kimchy recently changed the routing allocation code to
> treat an empty string as not set -- hoping that alleviates the
> problem. It would seem there has to be some way to remove the i.r.a
> settings on this index (maybe the java api? I haven't checked it out
> at all yet).
>
> Thanks,
> -james
>
> --
--
Shairon Toledo
> **[Microsoft Developer News | C++,C# & .NET Framework & More | CodeGuru](http://hashcode.me)**
>
> 您是否想要找关于幸运飞行艇 官方开奖历史记录 168飞艇全国统一开奖数据 还有幸运飞行艇官方开奖现场直播 幸运168飞艇全天精准稳定计划 这里都能一一提供。CodeGuru is where developers come to share ideas, articles, questions, answers, tips, tricks, comments, downloads, and more related to programming in areas including C++,...
--
---
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [November 1, 2012, 9:26pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/8 "2012-11-01T21:26:18Z")
On Thu, Nov 1, 2012 at 5:18 PM, Shairon Toledo [shairon.toledo@gmail.com](mailto:shairon.toledo@gmail.com) wrote:
> curl -XPUT location:9200/index\_name/\_settings -d '{
> "index.routing.allocation.include.\_id" : "eBSlwQ2QRNu4hs1mcjWGSQ"
> }'
OK, using \_id at the cluster level did nothing, but at the index
level, shards started relocating, and the replicas all started!
Not a permanent fix, but it seems there is a bug in the routing
allocation. I really want to get those settings out of this index.
--
---
### Author:  [@James\_Bardin](https://discuss.elastic.co/u/James_Bardin)
#### Post date: [November 2, 2012, 5:33pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/9 "2012-11-02T17:33:56Z")
So 0.19.11 has mitigated the problem for us. I'm guessing issue #2229
[1] is what really helped, in that empty strings are now ignored. New
replicas are allocated as expected.
1. [https://github.com/elasticsearch/elasticsearch/issues/2229](https://github.com/elasticsearch/elasticsearch/issues/2229)
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [November 2, 2012, 6:00pm UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/10 "2012-11-02T18:00:03Z")
I missed this thread while I was debugging the same issues:
[https://groups.google.com/d/msg/elasticsearch/6aGAUDtNtWw/48RZW9YRZ1QJ](https://groups.google.com/d/msg/elasticsearch/6aGAUDtNtWw/48RZW9YRZ1QJ)
No allocation settings are working for me. Cluster is unusable as-is since
shards can no longer be allocated. The whole point of my tests was to do a
rolling upgrade, but now it appears that a full cluster restart is needed.
A bit extreme.
--
Ivan
On Fri, Nov 2, 2012 at 10:33 AM, James Bardin [j.bardin@gmail.com](mailto:j.bardin@gmail.com) wrote:
> So 0.19.11 has mitigated the problem for us. I'm guessing issue #2229
> [1] is what really helped, in that empty strings are now ignored. New
> replicas are allocated as expected.
>
> 1. [Shard Allocation: `index.routing.allocation....` settings do not "remove" the setting on empty string · Issue #2229 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2229)
>
> --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:06am UTC](https://discuss.elastic.co/t/unassigned-replica-shards-and-an-unused-node/9508/11 "2017-07-06T03:06:24Z")