# Elasticsearch Shard Allocation - ALLOCATION\_FAILED due to apparent disk quota issues, nowhere near max **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [July 21, 2020, 2:07pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-allocation-allocation-failed-due-to-apparent-disk-quota-issues-nowhere-near-max/242038 "2020-07-21T14:07:47Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![DougR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dougr/32/48095_2.png) [@DougR](https://discuss.elastic.co/u/DougR)\ **Post date:** [July 21, 2020, 2:07pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-allocation-allocation-failed-due-to-apparent-disk-quota-issues-nowhere-near-max/242038/1 "2020-07-21T14:07:47Z")
# Environment - Elasticsearch 7.8.0 running in EKS - EFS filesystem - 3 master nodes - 5 data nodes - 649 total shards # TL;DR Receiving `ALLOCATION_FAILED` on shards due to apparent disk quota issues, should have more than enough disk space, also not exceeding max number of shards per data node, AFAIK. # Full Story I'm having an issue with unallocated replica shards. I'm receiving an error of "Disk quota exceeded." I'm using EFS and am showing 8E disk space available. I currently have a total of 649 shards spread among 5 data nodes. I executed this query: ```json GET /_cluster/allocation/explain { "index": ".monitoring-kibana-7-2020.07.21", "shard": 0, "primary": false } ``` ...and got the following results: ```json { "index" : ".monitoring-kibana-7-2020.07.21", "shard" : 0, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2020-07-21T13:26:31.961Z", "failed_allocation_attempts" : 5, "details" : "failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "Ku69PlSpQ8miS0CTrGYMXQ", "node_name" : "elk-es-data-4", "transport_address" : "10.229.27.241:9300", "node_attributes" : { "ilm_phase" : "hot", "ml.machine_memory" : "8589934592", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-07-21T13:26:31.961Z], failed_attempts[5], failed_nodes[[KwJM0xwQQMC2rrs4zxfqvw, Ku69PlSpQ8miS0CTrGYMXQ]], delayed=false, details[failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ], allocation_status[no_attempt]]]" } ] }, { "node_id" : "KwJM0xwQQMC2rrs4zxfqvw", "node_name" : "elk-es-data-0", "transport_address" : "10.229.25.185:9300", "node_attributes" : { "ilm_phase" : "hot", "ml.machine_memory" : "8589934592", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-07-21T13:26:31.961Z], failed_attempts[5], failed_nodes[[KwJM0xwQQMC2rrs4zxfqvw, Ku69PlSpQ8miS0CTrGYMXQ]], delayed=false, details[failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ], allocation_status[no_attempt]]]" } ] }, { "node_id" : "co-W7AfbSwWmABjAtw1LnQ", "node_name" : "elk-es-data-2", "transport_address" : "10.229.11.165:9300", "node_attributes" : { "ilm_phase" : "hot", "ml.machine_memory" : "8589934592", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-07-21T13:26:31.961Z], failed_attempts[5], failed_nodes[[KwJM0xwQQMC2rrs4zxfqvw, Ku69PlSpQ8miS0CTrGYMXQ]], delayed=false, details[failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ], allocation_status[no_attempt]]]" } ] }, { "node_id" : "hKKp7AOiRAKAI5EnB-83sQ", "node_name" : "elk-es-data-1", "transport_address" : "10.229.4.130:9300", "node_attributes" : { "ilm_phase" : "hot", "ml.machine_memory" : "8589934592", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-07-21T13:26:31.961Z], failed_attempts[5], failed_nodes[[KwJM0xwQQMC2rrs4zxfqvw, Ku69PlSpQ8miS0CTrGYMXQ]], delayed=false, details[failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ], allocation_status[no_attempt]]]" } ] }, { "node_id" : "heV_9oFRRQy6_VWNiFhaBg", "node_name" : "elk-es-data-3", "transport_address" : "10.229.16.67:9300", "node_attributes" : { "ilm_phase" : "hot", "ml.machine_memory" : "8589934592", "ml.max_open_jobs" : "20", "xpack.installed" : "true", "transform.node" : "true" }, "node_decision" : "no", "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2020-07-21T13:26:31.961Z], failed_attempts[5], failed_nodes[[KwJM0xwQQMC2rrs4zxfqvw, Ku69PlSpQ8miS0CTrGYMXQ]], delayed=false, details[failed shard on node [Ku69PlSpQ8miS0CTrGYMXQ]: failed recovery, failure RecoveryFailedException[[.monitoring-kibana-7-2020.07.21][0]: Recovery failed from {elk-es-data-3}{heV_9oFRRQy6_VWNiFhaBg}{AThiFSVtSWye8ppytYCLiA}{10.229.16.67}{10.229.16.67:9300}{dilrt}{ilm_phase=hot, ml.machine_memory=8589934592, ml.max_open_jobs=20, xpack.installed=true, transform.node=true} into {elk-es-data-4}{Ku69PlSpQ8miS0CTrGYMXQ}{BlN9xVdwRjaNphDHFW0PwQ}{10.229.27.241}{10.229.27.241:9300}{dilrt}{ml.machine_memory=8589934592, xpack.installed=true, transform.node=true, ilm_phase=hot, ml.max_open_jobs=20}]; nested: RemoteTransportException[[elk-es-data-3][10.229.16.67:9300][internal:index/shard/recovery/start_recovery]]; nested: RemoteTransportException[[elk-es-data-4][10.229.27.241:9300][internal:index/shard/recovery/clean_files]]; nested: IOException[Disk quota exceeded]; ], allocation_status[no_attempt]]]" }, { "decider" : "same_shard", "decision" : "NO", "explanation" : "a copy of this shard is already allocated to this node [[.monitoring-kibana-7-2020.07.21][0], node[heV_9oFRRQy6_VWNiFhaBg], [P], s[STARTED], a[id=l7wnPB-ORFOAyw62WvMIoA]]" } ] } ] } ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 21, 2020, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-allocation-allocation-failed-due-to-apparent-disk-quota-issues-nowhere-near-max/242038/2 "2020-07-21T14:42:37Z")
`Disk quota exceeded` indicates that a [write (or similar) syscall returned `EDQUOT`](https://linux.die.net/man/2/write). In other words, this error is coming from the operating system and Elasticsearch is simply reporting it verbatim. Note that this is different from `ENOSPC`: you can have ample free space on a disk and still exceed some quota or other. If this isn't a quota you've configured then you'll need to speak with AWS support to identify which limit you're hitting. Note that the reference manual [recommends against using EFS](https://www.elastic.co/guide/en/elasticsearch/plugins/7.8/cloud-aws-best-practices.html) for your data nodes. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 18, 2020, 2:52pm UTC](https://discuss.elastic.co/t/elasticsearch-shard-allocation-allocation-failed-due-to-apparent-disk-quota-issues-nowhere-near-max/242038/3 "2020-08-18T14:52:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.