# Why are some shards unassigned even though there are empty nodes?

**URL:** https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399
**Category:** Elasticsearch
**Created:** [December 8, 2016, 7:20am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399 "2016-12-08T07:20:40Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![WoJ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woj/32/4518_2.png) [@WoJ](https://discuss.elastic.co/u/WoJ)
#### Post date: [December 8, 2016, 7:20am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/1 "2016-12-08T07:20:40Z")

</div>

Hello

Looking today at my elasticsearch cluster (2.4.2) I found out that there are unallocated shards. There seems to be a pattern:

[![enter image description here](https://i.stack.imgur.com/vZmjW.png)](https://i.stack.imgur.com/vZmjW.png)

It looks like the shards which are on `eu2` are not allocated anywhere else (namely neither on `eu3`nor on `eu4`).

**What can be the reason for that?**

I did not fine-tune any allocation strategies, the machines are simply in a cluster with default settings and I do not see any error or warning in the logs (`/var/log/elasticsearch/<cluster name>.log`). While I did not look at the status for some time, it used to be green (with the same configuration).

I tried to look at the reason for the lack of assignment:

```
[root:~]# curl "http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason"| grep UNASSIGNED
veaees1.html 1 r UNASSIGNED NODE_LEFT
veaees1.html 0 r UNASSIGNED NODE_LEFT
eangvn1.html 1 r UNASSIGNED NODE_LEFT
eangvn1.html 0 r UNASSIGNED NODE_LEFT
wnoaog1.html 0 r UNASSIGNED NODE_LEFT
logstash-2015.01.28 0 r UNASSIGNED NODE_LEFT
logstash-2015.01.29 0 r UNASSIGNED NODE_LEFT
spsnwr1.html 1 r UNASSIGNED NODE_LEFT
spsnwr1.html 0 r UNASSIGNED NODE_LEFT
flex2gateway 0 r UNASSIGNED NODE_LEFT
logstash-2015.01.30 1 r UNASSIGNED NODE_LEFT
pnopsg1.html 0 r UNASSIGNED NODE_LEFT
nessus-candidates-2016-12-05 2 r UNASSIGNED INDEX_CREATED
(similar lines repeated - cut off due to lack of space for the question)
nessus-candidates-2016-12-07 0 r UNASSIGNED INDEX_CREATED
index 0 r UNASSIGNED NODE_LEFT
rwsnsr1.html 0 r UNASSIGNED NODE_LEFT
nessus-candidates-2016-12-01 2 r UNASSIGNED INDEX_CREATED
(...)
nessus-candidates-2016-12-04 2 r UNASSIGNED INDEX_CREATED
nessus_logs 0 r UNASSIGNED NODE_LEFT
spipe 0 r UNASSIGNED NODE_LEFT
phppath 0 r UNASSIGNED NODE_LEFT
ogrnge1.html 0 r UNASSIGNED NODE_LEFT
nessus 1 r UNASSIGNED NODE_LEFT
webui 0 r UNASSIGNED NODE_LEFT
rossae1.html 1 r UNASSIGNED NODE_LEFT
rossae1.html 0 r UNASSIGNED NODE_LEFT
nvseeo1.html 0 r UNASSIGNED NODE_LEFT
perl 0 r UNASSIGNED NODE_LEFT
dummy_index 1 r UNASSIGNED NODE_LEFT
dummy_index 0 r UNASSIGNED NODE_LEFT
enarow1.html 0 r UNASSIGNED NODE_LEFT
wvarrr1.html 2 r UNASSIGNED NODE_LEFT
wvarrr1.html 0 r UNASSIGNED NODE_LEFT
gawvoo1.html 2 r UNASSIGNED NODE_LEFT
gawvoo1.html 0 r UNASSIGNED NODE_LEFT
logs 1 r UNASSIGNED NODE_LEFT
logs 0 r UNASSIGNED NODE_LEFT
gnopgn1.html 1 r UNASSIGNED NODE_LEFT
(...)

```

I had to restart `eu4` for unrelated reasons - I had 15 unassigned shards before the reboot, it is now 62. All the unassigned shards _before_ the reboot had the `INDEX_CREATED` reason. _After_ rebooting, the new unassigned have the `NODE_LEFT` reason.

**EDIT** : I added a new node to the cluster to check if this would help. It is visible in the cluster but does not get any shards allocated (the unassigned shards remain unassigned and the new node is empty)

[![enter image description here](https://i.stack.imgur.com/qaAMg.png)](https://i.stack.imgur.com/qaAMg.png)

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 8, 2016, 7:48am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/2 "2016-12-08T07:48:39Z")

</div>

Did you disable allocation?

Check `_cluster/settings`

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [December 8, 2016, 7:52am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/3 "2016-12-08T07:52:02Z")

</div>

Are all nodes using the same Elasticsearch version?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 8, 2016, 7:52am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/4 "2016-12-08T07:52:50Z")

</div>

And do they have enough disk free.

---

<div class="post-metadata">

### Author: ![WoJ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/woj/32/4518_2.png) [@WoJ](https://discuss.elastic.co/u/WoJ)
#### Post date: [December 8, 2016, 9:08am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/5 "2016-12-08T09:08:02Z")

</div>

@Christian_Dahlqvist @warkolm

EDIT: my apologies - they indeed run the same **release and major** version, but the minor one was different (2.4.1 vs 2.4.2). after upgrading everything is now fine

* * *

The allocation is not disabled:

```
{
   "persistent": {},
   "transient": {
      "cluster": {
         "routing": {
            "allocation": {
               "enable": "true",
               "disable_allocation": "false"
            }
         }
      }
   }
}  

```

~~All the nodes run the same version of elasticsearch~~ and all have ample disk space.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 5, 2017, 9:08am UTC](https://discuss.elastic.co/t/why-are-some-shards-unassigned-even-though-there-are-empty-nodes/68399/6 "2017-01-05T09:08:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
