# Cannot get failed shard back online
**URL:** https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003
**Category:** Elasticsearch
**Created:** [June 4, 2021, 6:29pm UTC](https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003 "2021-06-04T18:29:04Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@agKaspar](https://discuss.elastic.co/u/agKaspar)
#### Post date: [June 4, 2021, 6:29pm UTC](https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003/1 "2021-06-04T18:29:04Z")
Hi, I'm trying to get failed shard in a single node cluster back online. The cluster is a single docker container running the `docker.elastic.co/elasticsearch/elasticsearch:7.9.2` image. It has 10 indices and somewhere this afternoon one index went red with the following error:
```auto
nested: IOException[failed engine (reason: [refresh failed source[write indexing buffer]])];
nested: CorruptIndexException[checksum status indeterminate: unexpected exception (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/xFpiL2YWSzOCBE7eNrGomQ/0/index/_zu_1.fnm")))];
nested: IOException[read past EOF: NIOFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/xFpiL2YWSzOCBE7eNrGomQ/0/index/_zu_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 6820: NIOFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/xFpiL2YWSzOCBE7eNrGomQ/0/index/_zu_1.fnm")];
nested: EOFException[read past EOF: NIOFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/indices/xFpiL2YWSzOCBE7eNrGomQ/0/index/_zu_1.fnm") buffer: java.nio.HeapByteBuffer[pos=0 lim=1024 cap=1024] chunkLen: 1024 end: 6820];
```
Searching online I found I should give this command a try:
```auto
/usr/share/elasticsearch/jdk/bin/java -cp /usr/share/elasticsearch/lib/lucene-core-8.6.2.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /usr/share/elasticsearch/data/nodes/0/indices/xFpiL2YWSzOCBE7eNrGomQ/0/index/
```
This seems to check 25 segment files (?) which takes about ~15min but then reports everything is fine.
I tried calling this endpoint:
```auto
POST _cluster/reroute?master_timeout=5m
{
"commands": [
{
"allocate_empty_primary": {
"index": "dossiers-en",
"shard": 0,
"node": "elastic-search-7cb7cf9bf8-dhwmn",
"accept_data_loss": true
}
}
]
}
```
But this gives me the same error as I started with again. Also restarting the whole container results in the same error.
Any tips on getting this index back up and running, preferably with no or minimal data loss?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 7, 2021, 4:20am UTC](https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003/2 "2021-06-07T04:20:32Z")
What's the output from `_cat/allocation?v` and an [allocation explain](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-allocation-explain.html) for the shard?
Also I'd suggest upgrading 7.13 is the latest and you're a few versions behind 🙂
---
### Author:  [@agKaspar](https://discuss.elastic.co/u/agKaspar)
#### Post date: [June 8, 2021, 6:33am UTC](https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003/3 "2021-06-08T06:33:17Z")
I was able to fix this by running `elasticsearch-shard remove-corrupt-data --dir ` on the faulty shard. The root cause most likely was poor kubernetes resource settings, causing the single elastic search pod to become evicted due to memory and/or CPU pressure. I've corrected this and the problem hasn't returned since.
We're in the process of replacing this single-node with a more advanced kubernetes deployment using the 'Elastic cloud on kubernetes' custom providers and guidelines ([link](https://www.elastic.co/downloads/elastic-cloud-kubernetes)), which should fix this problem for good.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2021, 6:34am UTC](https://discuss.elastic.co/t/cannot-get-failed-shard-back-online/275003/4 "2021-07-06T06:34:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.