### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 30, 2019, 12:14pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/1 "2019-07-30T12:14:31Z")
Hello Guys,
first off, our Cluster:
Master (Kibana)
-8GB
-Appache Web Server
-Does not filter/query
Workernode#01/#02
-32GB Ram each
-No Kibana
Soo my cluster is running about fine, there's just this thing that the Health under "Monitoring" is Yellow, since i have around 50% unassigned shards. What do i have to do about it? My predecessor set the default Shards to:
-5 Primary Shards
-1 replica
Which is way to much if i understand this right i should go with 2 Primary and 1 replica, but i struggle to find where i can fix this.
Any other Ideas why there are so many unassigned shards?
Greetz
Mo
---
### Author:  [@rugenl](https://discuss.elastic.co/u/rugenl)
#### Post date: [July 30, 2019, 12:30pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/2 "2019-07-30T12:30:05Z")
That should work even with 5 shards and 1 replica.
It could be something like node.attr's, like rack awareness. Do you see any node.attr options in elastisearch.yml? If so, what are the settings on both nodes?
Pick one index with unassigned shards, make sure it doesn't have replica = 2 or more.
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 30, 2019, 12:40pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/3 "2019-07-30T12:40:07Z")
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
Well i guess it's left as default ? It's like this on both sides.
---
### Author:  [@Silen\_logs](https://discuss.elastic.co/u/Silen_logs)
#### Post date: [July 30, 2019, 6:42pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/4 "2019-07-30T18:42:39Z")
Hi,
Did you read the logs ? When a shard cant be assign its possible that something has happened to these shards or with the cluster in the allocation moment.
Also, you can see which shards dont have been assigned with the command below:
CLI: `curl -XGET ":9200/_cat/shards" | grep "UNASSIGNED"`
DEV Tools: `GET /_cat/shards`
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 30, 2019, 6:52pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/5 "2019-07-30T18:52:17Z")
There are many reasons why shards might be unassigned, and it is rare for the logs to contain much useful information. The right way to diagnose the reason for an unassigned shard is to use the [allocation explain API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html).
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 31, 2019, 6:25am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/6 "2019-07-31T06:25:01Z")
@Silen_logs @DavidTurner First of all, thanks both of you for the hints & tips.
After using the allocation explain API, i got various reasons back why the shards cannot be assigned:
the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [13.226582219832771%] --\> I've upgraded the Disk with 250gb more, and set the Watermark rules to 80%.
the shard cannot be allocated to the same node on which a copy of the shard already exists --\> i don't really know what to about that? Any clues?
cannot allocate because allocation is not permitted to any of the nodes --\> I guess i'll find this in some .yml /conf file to change?
Thanks for your help guys, appreciate it!
Greetz
Moritz
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 31, 2019, 6:30am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/7 "2019-07-31T06:30:08Z")
Are all your Elasticsearch nodes running exactly the same version?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 31, 2019, 6:36am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/8 "2019-07-31T06:36:16Z")
It's quite hard to help from just the few small messages that you've picked out by hand. Please share the whole output.
> [@Moritz\_Kiesewetter](#):
>
> the shard cannot be allocated to the same node on which a copy of the shard already exists --\> i don't really know what to about that? Any clues?
That's normal. Elasticsearch won't put more than one copy of a shard on each node.
> [@Moritz\_Kiesewetter](#):
>
> the node is above the low watermark cluster setting
Ok, add more disk space, delete some data, or adjust the watermark settings. NB `cluster.routing.allocation.disk.watermark.low=85%` means the limit is when the disk is 85% full, so reducing the watermark to 80% is making the problem worse.
> [@Moritz\_Kiesewetter](#):
>
> cannot allocate because allocation is not permitted to any of the nodes
This is a summary of the more detailed information elsewhere in the output.
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 31, 2019, 6:47am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/9 "2019-07-31T06:47:32Z")
Ok, here is the whole output..
```auto
{
"index" : "filebeat-6.8.1-2019.07.30",
"shard" : 1,
"primary" : false,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "NODE_LEFT",
"at" : "2019-07-30T11:37:45.122Z",
"details" : "node_left [LZyAKlAfS-mNxvdjuaEwUg]",
"last_allocation_status" : "no_attempt"
},
"can_allocate" : "no",
"allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes",
"node_allocation_decisions" : [
{
"node_id" : "LZyAKlAfS-mNxvdjuaEwUg",
"node_name" : "LZyAKlA",
"transport_address" : "X.X.X.223:9300",
"node_attributes" : {
"ml.machine_memory" : "25111109632",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "same_shard",
"decision" : "NO",
"explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[filebeat-6.8.1-2019.07.30][1], node[LZyAKlAfS-mNxvdjuaEwUg], [P], s[STARTED], a[id=bW1xHKZcRPOlDzLLDDEKXA]]"
},
{
"decider" : "disk_threshold",
"decision" : "NO",
"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [14.997178982365764%]"
}
]
},
{
"node_id" : "LdEwqP_YRiWJRWA1_UyIgg",
"node_name" : "LdEwqP_",
"transport_address" : "X.X.X.222:9300",
"node_attributes" : {
"ml.machine_memory" : "25111093248",
"ml.max_open_jobs" : "20",
"xpack.installed" : "true",
"ml.enabled" : "true"
},
"node_decision" : "no",
"deciders" : [
{
"decider" : "disk_threshold",
"decision" : "NO",
"explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [13.211407213469084%]"
}
]
}
]
}
```
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 31, 2019, 6:49am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/10 "2019-07-31T06:49:18Z")
Yes, on both nodes, it it Version 6.8.1
Node#01
Installed Packages
Name : elasticsearch
Arch : noarch
Version : 6.8.1
Release : 1
Size : 227 M
Repo : installed
From repo : elastic-6.x
Node#02
Installed Packages
Name : elasticsearch
Arch : noarch
Version : 6.8.1
Release : 1
Size : 227 M
Repo : installed
From repo : elastic-6.x
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 31, 2019, 6:53am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/11 "2019-07-31T06:53:54Z")
Thanks, that helps. I reformatted it for you using the `` button to make it easier to read.
You have two data nodes and their disks are both over 85% full, so no replicas can be allocated in this cluster.
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 31, 2019, 7:06am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/12 "2019-07-31T07:06:08Z")
Sorry about that! Will do \ in the future.
Thanks a lot, i'll upgrade the Disks and hope this Problem will clear out.
Will mark you answer properly by tomorrow!
Thanks again.
---
### Author:  [@Moritz\_Kiesewetter](https://discuss.elastic.co/u/Moritz_Kiesewetter)
#### Post date: [July 31, 2019, 11:22am UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/13 "2019-07-31T11:22:58Z")
Little Update : I noticed that ES had troubles with allocating replica shards, because it thought that exact the same replica would already be on that node. Reassigned the Replicas (long process) and right now i'm at 200 unassigned shards, and it's going down with every index checked. I don't know yet if this is the final solution, since i have not improved my disk space yet.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 28, 2019, 12:28pm UTC](https://discuss.elastic.co/t/unassigned-shards-what-to-do-about-it/192892/15 "2019-08-28T12:28:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.