# ALLOCATION\_FAILED Clone Index **URL:** \ **Category:** Elasticsearch\ **Created:** [February 28, 2024, 3:59am UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289 "2024-02-28T03:59:44Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![hmacarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hmacarthur/32/137182_2.png) [@hmacarthur](https://discuss.elastic.co/u/hmacarthur)\ **Post date:** [February 28, 2024, 3:59am UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/1 "2024-02-28T03:59:44Z")
I am trying to use the /\_clone API in order to clone an Index. Before I clone my current index, I set `"read_only": "true"` as instructed by documentation. I then perform the clone operation and then as soon as the API call to the clone operation succeeds, I move both the new and old index to `"read_only": "false"` I do this once a day in the evening, and have been seeing my cluster drop to a RED status. After checking the logs, it seems like this issue is due to one of my 5 primary shards being unassigned. The primary shard has an error of `ALLOCATION_FAILED`. The full error message after calling `GET /_cat/shards/` is ```auto { "index" : "transactions_1709083815105", "shard" : 1, "primary" : true, "current_state" : "unassigned", "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2024-02-28T01:31:04.433Z", "failed_allocation_attempts" : 5, "details" : "failed shard on node [2adetjU2SM-DBPTDdGFh7w]: failed recovery, failure RecoveryFailedException[[transactions_1709083815105][1]: Recovery failed on {dd1352ff268360d1808ddc0395d50ead}{2adetjU2SM-DBPTDdGFh7w}{RaKibgTtQu2rleRUv9VTRg}{10.212.24.21}{10.212.24.21:9300}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=true, zone=us-east-1a, cross_cluster_transport_address=2600:1f18:7c4c:4581:40f7:ba0b:d426:6874, awareness_features_enabled=true, global_cpu_usage_ac_supported=true, shard_indexing_pressure_enabled=true, di_number=0, search_backpressure_feature_present=true}]; nested: ClusterBlockException[index [transactions_1709083815105] blocked by: [FORBIDDEN/5/index read-only (api)];]; ", "last_allocation_status" : "no" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [ { "node_id" : "2adetjU2SM-DBPTDdGFh7w", "node_name" : "dd1352ff268360d1808ddc0395d50ead", "node_decision" : "no", "weight_ranking" : 1, "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-28T01:31:04.433Z], failed_attempts[5], failed_nodes[[2adetjU2SM-DBPTDdGFh7w]], delayed=false, details[failed shard on node [2adetjU2SM-DBPTDdGFh7w]: failed recovery, failure RecoveryFailedException[[transactions_1709083815105][1]: Recovery failed on {dd1352ff268360d1808ddc0395d50ead}{2adetjU2SM-DBPTDdGFh7w}{RaKibgTtQu2rleRUv9VTRg}{10.212.24.21}{10.212.24.21:9300}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=true, zone=us-east-1a, cross_cluster_transport_address=2600:1f18:7c4c:4581:40f7:ba0b:d426:6874, awareness_features_enabled=true, global_cpu_usage_ac_supported=true, shard_indexing_pressure_enabled=true, di_number=0, search_backpressure_feature_present=true}]; nested: ClusterBlockException[index [transactions_1709083815105] blocked by: [FORBIDDEN/5/index read-only (api)];]; ], allocation_status[deciders_no]]]" } ] }, { "node_id" : "keC6H8fGS_6dX0l2HGGeNA", "node_name" : "b024b85c4243d6cb1b107d2d68005ceb", "node_decision" : "no", "weight_ranking" : 2, "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-28T01:31:04.433Z], failed_attempts[5], failed_nodes[[2adetjU2SM-DBPTDdGFh7w]], delayed=false, details[failed shard on node [2adetjU2SM-DBPTDdGFh7w]: failed recovery, failure RecoveryFailedException[[transactions_1709083815105][1]: Recovery failed on {dd1352ff268360d1808ddc0395d50ead}{2adetjU2SM-DBPTDdGFh7w}{RaKibgTtQu2rleRUv9VTRg}{10.212.24.21}{10.212.24.21:9300}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=true, zone=us-east-1a, cross_cluster_transport_address=2600:1f18:7c4c:4581:40f7:ba0b:d426:6874, awareness_features_enabled=true, global_cpu_usage_ac_supported=true, shard_indexing_pressure_enabled=true, di_number=0, search_backpressure_feature_present=true}]; nested: ClusterBlockException[index [transactions_1709083815105] blocked by: [FORBIDDEN/5/index read-only (api)];]; ], allocation_status[deciders_no]]]" }, { "decider" : "resize", "decision" : "NO", "explanation" : "source primary is allocated on another node" } ] }, { "node_id" : "FJ37FfInRjWJ-BiXWUR8kw", "node_name" : "4185113e6a8486abe34bf66e71cddd1a", "node_decision" : "no", "weight_ranking" : 3, "deciders" : [ { "decider" : "max_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2024-02-28T01:31:04.433Z], failed_attempts[5], failed_nodes[[2adetjU2SM-DBPTDdGFh7w]], delayed=false, details[failed shard on node [2adetjU2SM-DBPTDdGFh7w]: failed recovery, failure RecoveryFailedException[[transactions_1709083815105][1]: Recovery failed on {dd1352ff268360d1808ddc0395d50ead}{2adetjU2SM-DBPTDdGFh7w}{RaKibgTtQu2rleRUv9VTRg}{10.212.24.21}{10.212.24.21:9300}{dir}{dp_version=20210501, distributed_snapshot_deletion_enabled=false, cold_enabled=false, adv_sec_enabled=true, zone=us-east-1a, cross_cluster_transport_address=2600:1f18:7c4c:4581:40f7:ba0b:d426:6874, awareness_features_enabled=true, global_cpu_usage_ac_supported=true, shard_indexing_pressure_enabled=true, di_number=0, search_backpressure_feature_present=true}]; nested: ClusterBlockException[index [transactions_1709083815105] blocked by: [FORBIDDEN/5/index read-only (api)];]; ], allocation_status[deciders_no]]]" }, { "decider" : "resize", "decision" : "NO", "explanation" : "source primary is allocated on another node" } ] } ] } ``` I am confused why the shard allocation is failing due to ` [FORBIDDEN/5/index read-only (api)];];` My first hunch was due to the read-only toggling that I was doing, but since the other 4 shards succeeded, I figure something else is at play. My cluster free space is also around 800GB so I didn't think that the cluster was complaining about free storage space issues. In addition, this error happens intermittently. I run this code once a day, and I can reproduce it most of the time, but not all of the time. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 28, 2024, 6:59am UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/2 "2024-02-28T06:59:47Z")
Hi @hmacarthur and welcome! > [@hmacarthur](#): > > Before I clone my current index, I set > `"read_only": "true"` as instructed by documentation. The [clone index API docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-clone-index.html) say to do this: ```auto PUT /my_source_index/_settings { "settings": { "index.blocks.write": true } } ``` I would try that instead of setting `read_only`. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 28, 2024, 7:14am UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/3 "2024-02-28T07:14:20Z")
> [@hmacarthur](#): > > My cluster free space is also around 800GB so I didn't think that the cluster was complaining about free storage space issues. Elasticsearch by default looks at disk usage percentages for watermarks so if you have a very large disk 800GB left may have resulted in a watermark being exceeded and new shards not being able to allocate. There is also a limit on the number of shards per node that could prevent new shards from allocating. What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/cluster-stats.html)? Do you have any non-default settings for any of these parameters? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 28, 2024, 10:10pm UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/4 "2024-02-28T22:10:22Z")
> [@Christian\_Dahlqvist](#): > > Elasticsearch by default looks at disk usage percentages for watermarks so if you have a very large disk 800GB left may have resulted in a watermark being exceeded and new shards not being able to allocate. Since [8.5.0](https://github.com/elastic/elasticsearch/pull/88639) Elasticsearch handles very large disks better by using the greater of 85%/90%/95% and 200GiB/150GiB/100GiB for the low/high/flood-stage watermarks respectively. ---
**Author:** ![hmacarthur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hmacarthur/32/137182_2.png) [@hmacarthur](https://discuss.elastic.co/u/hmacarthur)\ **Post date:** [February 29, 2024, 4:50pm UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/5 "2024-02-29T16:50:37Z")
Thanks for getting back to me @DavidTurner @Christian_Dahlqvist It looks like @DavidTurner solution might be what I need, but I will continue to monitor how well it does over a few days and circle back. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 28, 2024, 4:51pm UTC](https://discuss.elastic.co/t/allocation-failed-clone-index/354289/6 "2024-03-28T16:51:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.