# \[allocate\_stale\_primary\] primary is already assigned **URL:** \ **Category:** Elasticsearch\ **Created:** [August 4, 2018, 12:15am UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995 "2018-08-04T00:15:15Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jethroft](https://avatars.discourse-cdn.com/v4/letter/j/e56c9b/32.png) [@jethroft](https://discuss.elastic.co/u/jethroft)\ **Post date:** [August 4, 2018, 12:15am UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/1 "2018-08-04T00:15:15Z")
My ES 5.6.3 cluster is yellow: ```auto administrator@srv4-sv:~$ curl -XGET 'localhost:9200/_cluster/health?pretty' { "cluster_name" : "redacted", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 15, "number_of_data_nodes" : 15, "active_primary_shards" : 172, "active_shards" : 350, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 1, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 99.71509971509973 } administrator@srv4-sv:~$ curl -sS -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED stats_new 1 r UNASSIGNED ALLOCATION_FAILED administrator@srv4-sv:~$ curl -sS -X POST 'localhost:9200/_cluster/allocation/explain?pretty' { "index" : "stats_new", "shard" : 1, "primary" : false, "current_state" : "unassigned", "unassigned_info" : { "reason" : "ALLOCATION_FAILED", "at" : "2018-08-03T22:38:21.312Z", "failed_allocation_attempts" : 5, "details" : "failed recovery, failure RecoveryFailedException[[stats_new][1]: Recovery failed from {srv4-sv}{w_6a_qIfTN2L8BN0wGiCyQ}{0KxMMAQETZywBTVNtriTkA}{10.64.2.17}{10.64.2.17:9300} into {srv4-ch}{7iSlk_IzRleE3AQag6cAfA}{i5S74TwfQeK lih5okbSjvQ}{10.64.3.17}{10.64.3.17:9300}]; nested: RemoteTransportException[[srv4-sv][10.64.2.17:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: IllegalStateException[try to recover [stats_new][1] from primary shard with sync id but number of docs differ: 29983940 (srv4-sv, primary) vs 29983938(srv4-ch)]; ", "last_allocation_status" : "no_attempt" }, "can_allocate" : "no", "allocate_explanation" : "cannot allocate because allocation is not permitted to any of the nodes", "node_allocation_decisions" : [...] } ``` According to the Internet, the fix is this: ```auto administrator@srv4-sv:~$ curl -sS -XPOST 'localhost:9200/_cluster/reroute' -d '{"commands":[{"allocate_stale_primary":{"index":"stats_new","shard":1,"node":"srv4-sv","accept_data_loss":true}}]}' ``` but that doesn't work: ```auto {"error":{"root_cause":[{"type":"remote_transport_exception","reason":"[srv6-sv][10.64.2.21:9300][cluster:admin/reroute]"}],"type":"illegal_argument_exception","reason":"[allocate_stale_primary] primary [stats_new][1] is already assigned"},"status":400} ``` Thoughts? ---
**Author:** ![jethroft](https://avatars.discourse-cdn.com/v4/letter/j/e56c9b/32.png) [@jethroft](https://discuss.elastic.co/u/jethroft)\ **Post date:** [August 6, 2018, 3:57pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/2 "2018-08-06T15:57:49Z")
(bump) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 6, 2018, 4:03pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/3 "2018-08-06T16:03:50Z")
Are all your nodes using exactly the same version of Elasticsearch (use the [cat nodes API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cat-nodes.html#_columns_2) to check)? ---
**Author:** ![jethroft](https://avatars.discourse-cdn.com/v4/letter/j/e56c9b/32.png) [@jethroft](https://discuss.elastic.co/u/jethroft)\ **Post date:** [August 6, 2018, 4:17pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/4 "2018-08-06T16:17:48Z")
Yes, they're all on 5.6.3 ---
**Author:** ![nhat](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nhat/32/22170_2.png) [@nhat](https://discuss.elastic.co/u/nhat)\ **Post date:** [August 6, 2018, 6:32pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/5 "2018-08-06T18:32:22Z")
@jethroft You need to remove the sync\_id, then retry allocation. 1. Issue a force flush to remove the sync-id: `POST /stats_new/_flush?force=true` 2. Issue a cluster reroute to retry allocation: `POST /_cluster/reroute?retry_failed` I hope this helps. ---
**Author:** ![jethroft](https://avatars.discourse-cdn.com/v4/letter/j/e56c9b/32.png) [@jethroft](https://discuss.elastic.co/u/jethroft)\ **Post date:** [August 6, 2018, 7:25pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/6 "2018-08-06T19:25:53Z")
@nhat Thanks. I tried that, but I don't think it did anything: ```auto $ curl -sS -XPOST 'localhost:9200/stats_new/_flush?force=true' {"_shards":{"total":10,"successful":9,"failed":0}} $ curl -sS -XPOST 'localhost:9200/_cluster/reroute?retry_failed' { "acknowledged": true, "state": { "version": 21335, "state_uuid": "WZNP_dwGQ1iXfpvKMfZHBg", "master_node": "-qnu-AcDSyyuSQZbZ_ykfQ", "blocks": {}, "nodes": { ... }, "routing_table": { "indices": { ... "stats_new": { "shards": { ... "1": [ { "state": "STARTED", "primary": true, "node": "w_6a_qIfTN2L8BN0wGiCyQ", "relocating_node": null, "shard": 1, "index": "stats_new", "allocation_id": { "id": "ikEbwB1lQPuBCLUeG96VzA" } }, { "state": "UNASSIGNED", "primary": false, "node": null, "relocating_node": null, "shard": 1, "index": "stats_new", "recovery_source": { "type": "PEER" }, "unassigned_info": { "reason": "ALLOCATION_FAILED", "at": "2018-08-03T22:38:21.312Z", "failed_attempts": 5, "delayed": false, "details": "failed recovery, failure RecoveryFailedException[[stats_new][1]: Recovery failed from {srv4-sv}{w_6a_qIfTN2L8BN0wGiCyQ}{0KxMMAQETZywBTVNtriTkA}{10.64.2.17}{10.64.2.17:9300} into {srv4-ch}{7iSlk_IzRleE3AQag6cAfA}{i5S74TwfQeKlih5okbSjvQ}{10.64.3.17}{10.64.3.17:9300}]; nested: RemoteTransportException[[srv4-sv][10.64.2.17:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: IllegalStateException[try to recover [stats_new][1] from primary shard with sync id but number of docs differ: 29983940 (srv4-sv, primary) vs 29983938(srv4-ch)]; ", "allocation_status": "no_attempt" } } ], ... } }, ... } }, "routing_nodes": { "unassigned": [ { "state": "UNASSIGNED", "primary": false, "node": null, "relocating_node": null, "shard": 1, "index": "stats_new", "recovery_source": { "type": "PEER" }, "unassigned_info": { "reason": "ALLOCATION_FAILED", "at": "2018-08-03T22:38:21.312Z", "failed_attempts": 5, "delayed": false, "details": "failed recovery, failure RecoveryFailedException[[stats_new][1]: Recovery failed from {srv4-sv}{w_6a_qIfTN2L8BN0wGiCyQ}{0KxMMAQETZywBTVNtriTkA}{10.64.2.17}{10.64.2.17:9300} into {srv4-ch}{7iSlk_IzRleE3AQag6cAfA}{i5S74TwfQeKlih5okbSjvQ}{10.64.3.17}{10.64.3.17:9300}]; nested: RemoteTransportException[[srv4-sv][10.64.2.17:9300][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[Phase[1] phase1 failed]; nested: RecoverFilesRecoveryException[Failed to transfer [0] files with total size of [0b]]; nested: IllegalStateException[try to recover [stats_new][1] from primary shard with sync id but number of docs differ: 29983940 (srv4-sv, primary) vs 29983938(srv4-ch)]; ", "allocation_status": "no_attempt" } } ], ... }, "snapshots": { "snapshots": [] } } } ``` ---
**Author:** ![jethroft](https://avatars.discourse-cdn.com/v4/letter/j/e56c9b/32.png) [@jethroft](https://discuss.elastic.co/u/jethroft)\ **Post date:** [August 6, 2018, 7:30pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/7 "2018-08-06T19:30:33Z")
Did it again with `retry_failed=true` (was missing `=true` before), now it's fixed. Thanks! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 3, 2018, 7:30pm UTC](https://discuss.elastic.co/t/allocate-stale-primary-primary-is-already-assigned/142995/8 "2018-09-03T19:30:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.