# Yellow Status- Unassigned Shards **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2019, 2:30pm UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272 "2019-09-05T14:30:51Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 5, 2019, 2:30pm UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/1 "2019-09-05T14:30:51Z")
Hi, I have many indices with yellow status and Unassigned Shards. How i can fixed that? Is this related that my nodes are on different countries and there is delay on network or it's something else? ![1](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f34e29c2a67db260390e0d01d7c63111691e2c7a.png) ---
**Author:** ![Preakness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preakness/32/35560_2.png) [@Preakness](https://discuss.elastic.co/u/Preakness)\ **Post date:** [September 5, 2019, 2:37pm UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/2 "2019-09-05T14:37:44Z")
Hello Antonopo, There are many reason for the cluster health to be yellow: 1. A node could have rejoined the cluster 2. Running out of disk space 3. Cluster restart etc Use the explain API to see the reason shards are not being assigned. ```auto curl -XGET localhost:9200/_cluster/allocation/explain?pretty ``` If you're running xpack security run it like this. ```auto curl -u jacknik:password -XGET https://localhost:9200/_cluster/allocation/explain?pretty ``` of course pass a username and password which is configured for your environment that has cluster management privileges. ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:22am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/3 "2019-09-06T06:22:56Z")
> [@Preakness](#): > > curl -XGET localhost:9200/\_cluster/allocation/explain?pretty { "index" : "metricbeat-7.0.1-2019.09.03", "shard" : 0, "primary" : false, "current\_state" : "unassigned", "unassigned\_info" : { "reason" : "ALLOCATION\_FAILED", "at" : "2019-09-03T00:02:10.320Z", "failed\_allocation\_attempts" : 5, "details" : "failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ", "last\_allocation\_status" : "no\_attempt" }, "can\_allocate" : "no", "allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node\_allocation\_decisions" : [ { "node\_id" : "GE4A2v29Qlybs6FyKbyCMw", "node\_name" : "xh-fr-elastic-2", "transport\_address" : "135.238.239.132:9300", "node\_attributes" : { "ml.machine\_memory" : "269930708992", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "disk\_threshold", "decision" : "NO", "explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [14.686823037524427%]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [nTGxqS2hTNe2O\_zh9Wx1tQ] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] }, ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:23am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/4 "2019-09-06T06:23:19Z")
{ "node\_id" : "TXQBZI3yRf69Q7CCJ2PdFQ", "node\_name" : "xh-it-elastic-1", "transport\_address" : "151.98.17.60:9300", "node\_attributes" : { "ml.machine\_memory" : "34359738368", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [nTGxqS2hTNe2O\_zh9Wx1tQ] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:23am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/5 "2019-09-06T06:23:35Z")
{ "node\_id" : "V-WUk1ZeQ7yROHALYmndkQ", "node\_name" : "xh-gr-elastic-2", "transport\_address" : "10.159.166.9:9300", "node\_attributes" : { "ml.machine\_memory" : "269930721280", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node\_concurrent\_incoming\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:23am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/6 "2019-09-06T06:23:51Z")
{ "node\_id" : "kZLf-LYfThiXOORoiECoaw", "node\_name" : "xh-gr-elastic-3", "transport\_address" : "10.158.67.107:9300", "node\_attributes" : { "ml.machine\_memory" : "17179332608", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [nTGxqS2hTNe2O\_zh9Wx1tQ] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:24am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/7 "2019-09-06T06:24:04Z")
{ "node\_id" : "nTGxqS2hTNe2O\_zh9Wx1tQ", "node\_name" : "xh-fr-elastic-1", "transport\_address" : "135.238.239.48:9300", "node\_attributes" : { "ml.machine\_memory" : "16654970880", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "same\_shard", "decision" : "NO", "explanation" : "the shard cannot be allocated to the same node on which a copy of the shard already exists [[metricbeat-7.0.1-2019.09.03][0], node[nTGxqS2hTNe2O\_zh9Wx1tQ], [P], s[STARTED], a[id=H2Km3EpqTJSU8efmVwFQhw]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [nTGxqS2hTNe2O\_zh9Wx1tQ] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:24am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/8 "2019-09-06T06:24:28Z")
{ "node\_id" : "rX7K8eJfQF6pRwt6i\_oUpA", "node\_name" : "xh-gr-elastic-1", "transport\_address" : "10.158.67.175:9300", "node\_attributes" : { "ml.machine\_memory" : "16654884864", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of outgoing shard recoveries [2] on the node [nTGxqS2hTNe2O\_zh9Wx1tQ] which holds the primary, cluster setting [cluster.routing.allocation.node\_concurrent\_outgoing\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:24am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/9 "2019-09-06T06:24:41Z")
{ "node\_id" : "vgsBYEfqSF6WOumwnVXFyQ", "node\_name" : "xh-it-elastic-2", "transport\_address" : "151.98.17.34:9300", "node\_attributes" : { "ml.machine\_memory" : "8186552320", "ml.max\_open\_jobs" : "20", "xpack.installed" : "true" }, "node\_decision" : "no", "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:02:10.320Z], failed\_attempts[5], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed to perform indices:data/write/bulk[s] on replica [metricbeat-7.0.1-2019.09.03][0], node[vgsBYEfqSF6WOumwnVXFyQ], [R], recovery\_source[peer recovery], s[INITIALIZING], a[id=6DLJqabRT02SxipKi25cFA], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2019-09-03T00:01:29.171Z], failed\_attempts[4], delayed=false, details[failed shard on node [vgsBYEfqSF6WOumwnVXFyQ]: failed recovery, failure RecoveryFailedException[[metricbeat-7.0.1-2019.09.03][0]: Recovery failed from {xh-fr-elastic-1}{nTGxqS2hTNe2O\_zh9Wx1tQ}{Jn2CA99oTqKHokKMcwdzmw}{135.238.239.48}{135.238.239.48:9300}{dim}{ml.machine\_memory=16654970880, ml.max\_open\_jobs=20, xpack.installed=true} into {xh-it-elastic-2}{vgsBYEfqSF6WOumwnVXFyQ}{yc1Cy7AsS-yY\_0Vv8ETeFg}{151.98.17.34}{151.98.17.34:9300}{dim}{ml.machine\_memory=8186552320, xpack.installed=true, ml.max\_open\_jobs=20}]; nested: RemoteTransportException[[xh-fr-elastic-1][135.238.239.48:9300][internal:index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException[Phase[1] prepare target for translog failed]; nested: RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][internal:index/shard/recovery/prepare\_translog]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8234652702/7.6gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8234652232/7.6gb], new bytes reserved: [470/470b], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=1814918/1.7mb, accounting=37715236/35.9mb]]; ], allocation\_status[no\_attempt]], failure RemoteTransportException[[xh-it-elastic-2][151.98.17.34:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [\] would be [8151435664/7.5gb], which is larger than the limit of [8094194073/7.5gb], real usage: [8151430896/7.5gb], new bytes reserved: [4768/4.6kb], usages [request=0/0b, fielddata=0/0b, in\_flight\_requests=13384/13kb, accounting=39491168/37.6mb]]; ], allocation\_status[no\_attempt]]]" }, { "decider" : "throttling", "decision" : "THROTTLE", "explanation" : "reached the limit of incoming shard recoveries [2], cluster setting [cluster.routing.allocation.node\_concurrent\_incoming\_recoveries=2] (can also be set via [cluster.routing.allocation.node\_concurrent\_recoveries])" } ] } ] } ---
**Author:** ![antonopo](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@antonopo](https://discuss.elastic.co/u/antonopo)\ **Post date:** [September 6, 2019, 6:24am UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/10 "2019-09-06T06:24:55Z")
I am getting all these ---
**Author:** ![Preakness](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/preakness/32/35560_2.png) [@Preakness](https://discuss.elastic.co/u/Preakness)\ **Post date:** [September 6, 2019, 1:21pm UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/11 "2019-09-06T13:21:17Z")
You're low on disk space, see this reason: > [@antonopo](#): > > "explanation" : "the node is above the low watermark cluster setting [cluster.routing.allocation.disk.watermark.low=85%], using more disk space than the maximum allowed [85.0%], actual free: [14.686823037524427%]" You'll need to remove data from Elasticsearch to get below 85% utilization then the shards will be assigned. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2019, 1:27pm UTC](https://discuss.elastic.co/t/yellow-status-unassigned-shards/198272/12 "2019-10-04T13:27:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.