# Reassigning Shards

**URL:** <https://discuss.elastic.co/t/reassigning-shards/48736>\
**Category:** Elasticsearch\
**Created:** [April 28, 2016, 9:07pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736 "2016-04-28T21:07:24Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 28, 2016, 9:07pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/1 "2016-04-28T21:07:24Z")

</div>

We had to hard shutdown all of our Elasticsearch server due to an environmental issue. Now all of our shards are unassigned:

```
{
  "cluster_name" : "Elasticsearch-Cluster-1",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 101,
  "active_shards" : 101,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 109,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 48.095238095238095
}

logstash-2016.04.27 1 r UNASSIGNED
logstash-2016.04.27 0 r UNASSIGNED
logstash-2016.04.28 2 p UNASSIGNED
logstash-2016.04.28 2 r UNASSIGNED
logstash-2016.04.28 1 p UNASSIGNED
logstash-2016.04.28 1 r UNASSIGNED
logstash-2016.04.28 0 p UNASSIGNED
logstash-2016.04.28 0 r UNASSIGNED
.marvel-es-2016.04.28 0 p UNASSIGNED
.marvel-es-2016.04.28 0 r UNASSIGNED
.marvel-es-data-1 0 r UNASSIGNED
.marvel-es-2016.04.25 0 r UNASSIGNED
.marvel-es-2016.04.24 0 r UNASSIGNED
.marvel-es-2016.04.27 0 r UNASSIGNED
.marvel-es-2016.04.26 0 r UNASSIGNED
logstash-2016.03.29 2 r UNASSIGNED
logstash-2016.03.29 1 r UNASSIGNED
logstash-2016.03.29 0 r UNASSIGNED
logstash-2016.03.28 2 r UNASSIGNED
logstash-2016.03.28 1 r UNASSIGNED
logstash-2016.03.28 0 r UNASSIGNED
.marvel-es-1-2016.04.28 0 r UNASSIGNED
.marvel-es-2016.03.30 0 r UNASSIGNED
.marvel-es-2016.03.31 0 r UNASSIGNED
.marvel-es-2016.04.01 0 r UNASSIGNED
.marvel-es-2016.03.28 0 r UNASSIGNED
.marvel-es-2016.03.29 0 r UNASSIGNED
logstash-2016.04.01 2 r UNASSIGNED
logstash-2016.04.01 1 r UNASSIGNED
logstash-2016.04.01 0 r UNASSIGNED
logstash-2016.04.02 2 r UNASSIGNED
logstash-2016.04.02 1 r UNASSIGNED
logstash-2016.04.02 0 r UNASSIGNED
logstash-2016.03.31 2 r UNASSIGNED
logstash-2016.03.31 1 r UNASSIGNED
logstash-2016.03.31 0 r UNASSIGNED
logstash-2016.04.03 2 r UNASSIGNED
logstash-2016.04.03 1 r UNASSIGNED
logstash-2016.04.03 0 r UNASSIGNED
logstash-2016.03.30 2 r UNASSIGNED
logstash-2016.03.30 1 r UNASSIGNED
logstash-2016.03.30 0 r UNASSIGNED
logstash-2016.04.04 2 r UNASSIGNED
logstash-2016.04.04 1 r UNASSIGNED
logstash-2016.04.04 0 r UNASSIGNED
logstash-2016.04.09 2 r UNASSIGNED
logstash-2016.04.09 1 r UNASSIGNED
logstash-2016.04.09 0 r UNASSIGNED
logstash-2016.04.05 2 r UNASSIGNED
logstash-2016.04.05 1 r UNASSIGNED
logstash-2016.04.05 0 r UNASSIGNED
logstash-2016.04.06 2 r UNASSIGNED
logstash-2016.04.06 1 r UNASSIGNED
logstash-2016.04.06 0 r UNASSIGNED
logstash-2016.04.07 2 r UNASSIGNED
logstash-2016.04.07 1 r UNASSIGNED
logstash-2016.04.07 0 r UNASSIGNED
logstash-2016.04.08 2 r UNASSIGNED
logstash-2016.04.08 1 r UNASSIGNED
logstash-2016.04.08 0 r UNASSIGNED
.marvel-es-2016.04.10 0 r UNASSIGNED
.marvel-es-2016.04.12 0 r UNASSIGNED
.marvel-es-2016.04.11 0 r UNASSIGNED
.marvel-es-2016.04.07 0 r UNASSIGNED
.marvel-es-2016.04.06 0 r UNASSIGNED
.marvel-es-2016.04.09 0 r UNASSIGNED
.marvel-es-2016.04.08 0 r UNASSIGNED
.kibana 0 r UNASSIGNED
.marvel-es-2016.04.03 0 r UNASSIGNED
.marvel-es-2016.04.02 0 r UNASSIGNED
.marvel-es-2016.04.05 0 r UNASSIGNED
.marvel-es-2016.04.04 0 r UNASSIGNED
logstash-2016.04.12 2 r UNASSIGNED
logstash-2016.04.12 1 r UNASSIGNED
logstash-2016.04.12 0 r UNASSIGNED
logstash-2016.04.13 2 r UNASSIGNED
logstash-2016.04.13 1 r UNASSIGNED
logstash-2016.04.13 0 r UNASSIGNED
logstash-2016.04.14 2 r UNASSIGNED
logstash-2016.04.14 1 r UNASSIGNED
logstash-2016.04.14 0 r UNASSIGNED

```

Anyone have any suggestions on how to best bring them back into the cluster?

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 28, 2016, 9:45pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/2 "2016-04-28T21:45:58Z")

</div>

I found the following command that is supposed to go thru each unassigned shard and assign it.

```
for shard in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{print $2}'); do
    curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "t37", 
                  "shard" : $shard, 
                  "node" : "datanode15", 
                  "allow_primary" : true
              }
            }
        ]
    }'
    sleep 5
done

```

But when I run the command I get the following error:

`{"error":{"root_cause":[{"type":"json_parse_exception","reason":"Unrecognized token '$shard': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty.ChannelBufferStreamInput@4351a6c2; line: 5, column: 36]"}],"type":"json_parse_exception","reason":"Unrecognized token '$shard': was expecting ('true', 'false' or 'null')\n at [Source: org.elasticsearch.transport.netty.ChannelBufferStreamInput@4351a6c2; line: 5, column: 36]"},"status":500}`

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 10:03am UTC](https://discuss.elastic.co/t/reassigning-shards/48736/3 "2016-04-29T10:03:06Z")

</div>

Hi! Look at data and active master logs are any error or warnings about shards?  
Another point, look at fs level on nodes if there shard's files or shards directory just empty?

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 10:47am UTC](https://discuss.elastic.co/t/reassigning-shards/48736/4 "2016-04-29T10:47:50Z")

</div>

> [@jnpetty](#):
>
> Unrecognized token '$shard'

There error in script as $shard does not recognized as variable. Just change `$shard` to `'$shard'`:

Try something like this (you need to parse NODE, INDEX, SHARD not only SHARD):

```
#!/bin/bash
for line in $(curl -XGET http://localhost:9200/_cat/shards | grep UNASSIGNED | awk '{printf "%s;%s;%s\n", $1, $2, $8}' ); do
    IFS=';' read index shard node <<< "$line"
    curl -XPOST 'localhost:9200/_cluster/reroute' -d '{
        "commands" : [ {
              "allocate" : {
                  "index" : "'$index'", 
                  "shard" : '$shard', 
                  "node" : "'$node'", 
                  "allow_primary" : true
              }
            }
        ]
    }'
    sleep 5
done

```

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 29, 2016, 3:09pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/5 "2016-04-29T15:09:59Z")

</div>

The struggle continues:

`{"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[p-es-1][192.168.56.86:9300][cluster:admin/reroute]"}],"type":"illegal_argument_exception","reason":"[allocate] allocation of [logstash-2016.04.20][2] on node {p-es-1}{uUw7qMvGSCqyILY1IbT9NQ}{192.168.56.86}{192.168.56.86:9300} is not allowed, reason: [YES(target node version [2.3.2] is same or newer than source node version [2.3.2])][NO(shard cannot be allocated on same node [uUw7qMvGSCqyILY1IbT9NQ] it already exists on)][YES(shard not primary or relocation disabled)][YES(total shard limit disabled: [index: -1, cluster: -1] <= 0)][NO(more than allowed [85.0%] used disk on node, free: [14.640832792774445%])][YES(below shard recovery limit of [2])][YES(primary is already active)][YES(no allocation awareness enabled)][YES(node passes include/exclude/require filters)][YES(allocation disabling is ignored)][YES(allocation disabling is ignored)]"},"status":400}`

The line that sticks out is: "shard cannot be allocated on same node [uUw7qMvGSCqyILY1IbT9NQ] it already exists on". I tried putting it on all of our hosts and ended up with the same error.

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 3:31pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/6 "2016-04-29T15:31:40Z")

</div>

Also pay attention to this message:

> [NO(more than allowed [85.0%] used disk on node, free: [14.640832792774445%])]

Maybe you hit by free space limit?

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 29, 2016, 3:35pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/7 "2016-04-29T15:35:22Z")

</div>

I dont believe so since the index says that I am trying to assign is only 800mb and the smallest free size is around 15gb

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 3:41pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/8 "2016-04-29T15:41:07Z")

</div>

What is the output of `df -h` ? As you can see there are percentage in message not absolute numbers, so no matter how much you have in compare to shard size, it's just less then 15% of free space.

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 3:45pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/9 "2016-04-29T15:45:47Z")

</div>

[https://www.elastic.co/guide/en/elasticsearch/reference/2.3/disk-allocator.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/disk-allocator.html)

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 29, 2016, 5:01pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/10 "2016-04-29T17:01:36Z")

</div>

Ok let me expand the drives and give it another shot.

---

<div class="post-metadata">

**Author:** ![rusty](https://avatars.discourse-cdn.com/v4/letter/r/f17d59/32.png) [@rusty](https://discuss.elastic.co/u/rusty)\
**Post date:** [April 29, 2016, 5:37pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/11 "2016-04-29T17:37:18Z")

</div>

In fact, if you are know data rate, you can completely disable: `cluster.routing.allocation.disk.threshold_enabled`  
or change `cluster.routing.allocation.disk.watermark.low` and `cluster.routing.allocation.disk.watermark.high` to higher percent values or to an absolute byte value (like 900mb).

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [April 29, 2016, 8:27pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/12 "2016-04-29T20:27:41Z")

</div>

I got it all resolved! Thanks for your help.

---

<div class="post-metadata">

**Author:** ![Ned](https://avatars.discourse-cdn.com/v4/letter/n/848f3c/32.png) [@Ned](https://discuss.elastic.co/u/Ned)\
**Post date:** [October 3, 2016, 4:12pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/13 "2016-10-03T16:12:21Z")

</div>

Hi Jnpetty, I seem to be experience the same issue as you did. What finally ended up working for you?

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [October 3, 2016, 4:40pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/14 "2016-10-03T16:40:13Z")

</div>

Hi Ned,

I actually didnt make any changes to the cluster configuration. I didn't realize that when the disk space gets below 15% shard allocation stops completely. In our case we still had several 100Gb free, but it was below 15%. So now I just monitor disk space a little closer and make sure indexes are being deleted. Once I rebuild the cluster in production I may set an absolute value based on our disk space.

---

<div class="post-metadata">

**Author:** ![Ned](https://avatars.discourse-cdn.com/v4/letter/n/848f3c/32.png) [@Ned](https://discuss.elastic.co/u/Ned)\
**Post date:** [October 4, 2016, 2:35am UTC](https://discuss.elastic.co/t/reassigning-shards/48736/15 "2016-10-04T02:35:31Z")

</div>

Gotcha, thanks for your quick response! I wonder if there is any danger in turning off the disk allocation decider.

---

<div class="post-metadata">

**Author:** ![jnpetty](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@jnpetty](https://discuss.elastic.co/u/jnpetty)\
**Post date:** [October 4, 2016, 3:28pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/16 "2016-10-04T15:28:34Z")

</div>

In theory as long as your data is on a different file-system, it probably wont make a difference. I think the setting is only in place to protect you from filling up you root file-system and halting your system.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/reassigning-shards/48736/17 "2017-07-05T22:15:05Z")

</div>