# Shard data is missing without any reason or log

**URL:** https://discuss.elastic.co/t/shard-data-is-missing-without-any-reason-or-log/161248
**Category:** Elasticsearch
**Created:** [December 18, 2018, 6:31am UTC](https://discuss.elastic.co/t/shard-data-is-missing-without-any-reason-or-log/161248 "2018-12-18T06:31:37Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![kamal](https://avatars.discourse-cdn.com/v4/letter/k/a698b9/32.png) [@kamal](https://discuss.elastic.co/u/kamal)
#### Post date: [December 18, 2018, 6:31am UTC](https://discuss.elastic.co/t/shard-data-is-missing-without-any-reason-or-log/161248/1 "2018-12-18T06:31:38Z")

</div>

Hi  
I was doing a huge indexing job(about 400 billion records), but suddenly one of the nodes went down because the power failed, after fixing the power, one shard was missing, so I used elasticsearch-trasnlog and every thing was fine for days (more than 100Billion new records was indexed), but suddenly faced a new problem:

failed shard, IndexShardRecoveryException, codec header mismach, acctual header=-6546052509 vs expcted header=1071082519 resource=mmapindexinput(path=.../index\_3jm.cfs))

I see the data of shard is not there!!!  
Relocate that shard and another shard failed, and I was forced to relocate the second shard to.  
Any one can give me a hint?

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [December 18, 2018, 8:19am UTC](https://discuss.elastic.co/t/shard-data-is-missing-without-any-reason-or-log/161248/3 "2018-12-18T08:19:15Z")

</div>

Relates to [Failed shard recovery after hard shutdown](https://discuss.elastic.co/t/failed-shard-recovery-after-hard-shutdown/160998).

It sounds like you have other lurking corruptions after your power outage. Unfortunately with zero replicas and storage that isn't resilient to powerloss there's no way to get out of this situation without losing yet more data. If it were me, I would start again.

Setting [`index.shard.check_on_startup: true`](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#_static_index_settings) will at least check the whole index for corruption at startup rather than waiting for a corruption to be detected during a merge. This'll take a while, and you should set it back to `null` afterwards otherwise that check will happen every time.

If you are using version ≥ 6.5 then [the `elasticsearch-shard` tool](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/shard-tool.html) will delete any corrupted segments, but as with `elasticsearch-translog` this entails arbitrary data loss.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 15, 2019, 8:19am UTC](https://discuss.elastic.co/t/shard-data-is-missing-without-any-reason-or-log/161248/4 "2019-01-15T08:19:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
