# Primary Shard Allocation\_Failed **URL:** \ **Category:** Elasticsearch\ **Created:** [September 23, 2022, 3:45pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033 "2022-09-23T15:45:30Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ivandbarria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivandbarria/32/111190_2.png) [@ivandbarria](https://discuss.elastic.co/u/ivandbarria)\ **Post date:** [September 23, 2022, 3:45pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/1 "2022-09-23T15:45:30Z")
Hello, Hope everyone is doing great. We had a power outage and our ES server went down. When the server went back online I noticed ES status on red: ```auto "cluster_name" : "elasticsearch", "status" : "red", "timed_out" : false, "number_of_nodes" : 1, "number_of_data_nodes" : 1, "active_primary_shards" : 5, "active_shards" : 5, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 26, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 16.129032258064516 ``` Checked status: ```auto index shard prirep state node unassigned.reason index_expedients 1 r UNASSIGNED CLUSTER_RECOVERED index_expedients 3 r UNASSIGNED CLUSTER_RECOVERED index_expedients 4 r UNASSIGNED CLUSTER_RECOVERED index_expedients 2 p UNASSIGNED ALLOCATION_FAILED index_expedients 2 r UNASSIGNED CLUSTER_RECOVERED index_expedients 0 r UNASSIGNED CLUSTER_RECOVERED index_customers 1 p UNASSIGNED ALLOCATION_FAILED index_customers 1 r UNASSIGNED CLUSTER_RECOVERED index_customers 3 p UNASSIGNED ALLOCATION_FAILED index_customers 3 r UNASSIGNED CLUSTER_RECOVERED index_customers 2 p UNASSIGNED ALLOCATION_FAILED index_customers 2 r UNASSIGNED CLUSTER_RECOVERED index_customers 4 p UNASSIGNED ALLOCATION_FAILED index_customers 4 r UNASSIGNED CLUSTER_RECOVERED index_customers 0 p UNASSIGNED ALLOCATION_FAILED index_customers 0 r UNASSIGNED CLUSTER_RECOVERED index_general 1 p UNASSIGNED ALLOCATION_FAILED index_general 1 r UNASSIGNED CLUSTER_RECOVERED index_general 3 p UNASSIGNED ALLOCATION_FAILED index_general 3 r UNASSIGNED CLUSTER_RECOVERED index_general 2 p UNASSIGNED ALLOCATION_FAILED index_general 2 r UNASSIGNED CLUSTER_RECOVERED index_general 4 p UNASSIGNED ALLOCATION_FAILED index_general 4 r UNASSIGNED CLUSTER_RECOVERED index_general 0 p UNASSIGNED ALLOCATION_FAILED index_general 0 r UNASSIGNED CLUSTER_RECOVERED index_expedients 1 p STARTED SPANBIWEB02 index_expedients 3 p STARTED SPANBIWEB02 index_expedients 4 p STARTED SPANBIWEB02 index_expedients 0 p STARTED SPANBIWEB02 .kibana 0 p STARTED SPANBIWEB02 ``` Im getting this error at the primary shards that have failed Allocation: ```auto { "index": "index_expedients", "shard": 2, "primary": true, "current_state": "unassigned", "unassigned_info": { "reason": "ALLOCATION_FAILED", "at": "2022-09-23T15:32:36.195Z", "failed_allocation_attempts": 5, "details": "failed shard on node [hfFii1P3QxKPvijusWAExA]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: EOFException[read past EOF. pos [1973] length: [4] end: [1973]]; ", "last_allocation_status": "no" }, "can_allocate": "no", "allocate_explanation": "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy", "node_allocation_decisions": [ { "node_id": "hfFii1P3QxKPvijusWAExA", "node_name": "SPANBIWEB02", "transport_address": "127.0.0.1:9300", "node_attributes": { "ml.machine_memory": "17179398144", "xpack.installed": "true", "ml.max_open_jobs": "20", "ml.enabled": "true" }, "node_decision": "no", "store": { "in_sync": true, "allocation_id": "G30Aru58QhWc7kY2hrNecw" }, "deciders": [ { "decider": "max_retry", "decision": "NO", "explanation": "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/_cluster/reroute?retry_failed=true] to retry, [unassigned_info[[reason=ALLOCATION_FAILED], at[2022-09-23T15:32:36.195Z], failed_attempts[5], delayed=false, details[failed shard on node [hfFii1P3QxKPvijusWAExA]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: EOFException[read past EOF. pos [1973] length: [4] end: [1973]]; ], allocation_status[deciders_no]]]" } ] } ] } ``` Unfortunately I don't have a snapshot to recover. Is there a way to recover these shards? Regards, ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 24, 2022, 7:35am UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/2 "2022-09-24T07:35:43Z")
> [@ivandbarria](#): > > `EngineException[failed to recover from translog]; nested: EOFException[read past EOF. pos [1973] length: [4] end: [1973]]; ", "last_allocation_status": "no"` This error means that your disks were behaving incorrectly: they confirmed to Elasticsearch that some data was written durably too soon, leading to that data being lost in the power outage. This is a pretty common trick on cheaper hardware since it makes it look like the disks are much faster than they actually are, although there's sometimes a way to configure them to behave properly. The reference manual has [some notes on troubleshooting disk issues](https://www.elastic.co/guide/en/elasticsearch/reference/current/corruption-troubleshooting.html) - in particular [`diskchecker.pl`](https://gist.github.com/bradfitz/3172656) should be able to find the same problem which will help you explore the possible fixes. Unfortunately it means that the data Elasticsearch wrote simply never made it to the disk, it's gone, there's no way to recover the missing portions. The best remedy is to restore from a snapshot. If your data matters to you, you really should be taking snapshots! You may be able to use [the `elasticsearch-shard` tool](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-tool.html) to recover some of the remaining data, although there's no guarantee how much might be lost in the process. ---
**Author:** ![ivandbarria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ivandbarria/32/111190_2.png) [@ivandbarria](https://discuss.elastic.co/u/ivandbarria)\ **Post date:** [September 26, 2022, 5:27pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/3 "2022-09-26T17:27:07Z")
Thank you David. Im not able to find elasticsearch\_ shard tool, Im using version 6.3.0 @DavidTurner Do you know if there´s a way to get elasticsearch\_ shard tool for version 6.3.0? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 26, 2022, 8:17pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/4 "2022-09-26T20:17:05Z")
No, sorry, I don't remember if there's a way to recover anything from this situation in such old versions. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 26, 2022, 9:32pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/5 "2022-09-26T21:32:23Z")
6.3 is super old and [EOL](https://www.elastic.co/support/eol), you should really look to upgrade ASAP. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 24, 2022, 9:33pm UTC](https://discuss.elastic.co/t/primary-shard-allocation-failed/315033/6 "2022-10-24T21:33:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.