# Shard recovery fails after resizing Google Cloud Platform's Persistent Disk

**URL:** https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936
**Category:** Elasticsearch
**Created:** [September 9, 2017, 3:14pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936 "2017-09-09T15:14:12Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 9, 2017, 3:14pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/1 "2017-09-09T15:14:13Z")

</div>

I am running Elasticsearch inside Kubernetes cluster on Google Cloud Platform. The index is stored on a persistent disk. [[Kubernetes Configuration](https://github.com/loklak/loklak_server/blob/development/kubernetes/yamls/api/elasticsearch/es-deployment.yaml)]

Elasticsearch version is `2.0.0`.

Recently, the disk was full and I resized it from 100GB to 200GB using Google Cloud Console. I stopped the Kubernetes service running ES -

```bash
kubectl delete -R -f /path/to/elasticsearch/configs

```

Then I created a compute engine instance and resized partition following [this](https://cloud.google.com/compute/docs/disks/add-persistent-disk#resize_partitions) document.

Then I recreated ES deployment on Kubernetes and was welcomed by this error -

```auto
I [2017-09-09 14:53:35,843][WARN][cluster.action.shard] [Controller] [messages_week][4] received shard failed for [messages_week][4], node[3cz-CBk5Ro-0MCy0A9cp6A], [P], v[137], s[INITIALIZING], a[id=2gM_hcRETP2Eg2vsAjJkyQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-09-09T14:52:56.492Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30]; ]], indexUUID [vhW2TG5uTHiYPUwNZRJSiQ], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30]; ]
 
I [messages_week][[messages_week][4]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30];
 
I at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:258)
 
I at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:60)
 
I at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:133)
 
I at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 
I at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 
I at java.lang.Thread.run(Thread.java:745)
 
I Caused by: [messages_week][[messages_week][4]] EngineCreationFailureException[failed to recover from translog]; nested: EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30];
 
I at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:157)
 
I at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
 
I at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1349)
 
I at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1344)
 
I at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:889)
 
I at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:866)
 
I at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:249)
 
I ... 5 more
 
I Caused by: [messages_week][[messages_week][4]] EngineException[failed to recover from translog]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30];
 
I at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:233)
 
I at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:154)
 
I ... 11 more
 
I Caused by: TranslogCorruptedException[translog corruption while reading from stream]; nested: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30];
 
I at org.elasticsearch.index.translog.Translog.readOperation(Translog.java:1620)
 
I at org.elasticsearch.index.translog.TranslogReader.read(TranslogReader.java:132)
 
I at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.readOperation(TranslogReader.java:299)
 
I at org.elasticsearch.index.translog.TranslogReader$ReaderSnapshot.next(TranslogReader.java:290)
 
I at org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:70)
 
I at org.elasticsearch.index.engine.InternalEngine.recoverFromTranslog(InternalEngine.java:219)
 
I ... 12 more
 
I Caused by: TranslogCorruptedException[translog stream is corrupted, expected: 0x64332db9, got: 0x74223a30]
 
I at org.elasticsearch.index.translog.Translog.verifyChecksum(Translog.java:1577)
 
I at org.elasticsearch.index.translog.Translog.readOperation(Translog.java:1610)
 
I ... 17 more

```

How can I resolve this state?

Thanks in advance.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 9, 2017, 9:31pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/2 "2017-09-09T21:31:05Z")

</div>

> [@singhpratyush](#):
>
> Elasticsearch version is 2.0.0.

You should really upgrade.

> [@singhpratyush](#):
>
> TranslogCorruptedException[translog stream is corrupted

Did you/can you run a filesystem check?

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 10, 2017, 4:24am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/3 "2017-09-10T04:24:45Z")

</div>

> [@warkolm](#):
>
> Did you/can you run a filesystem check?

I ran a filesystem check after resizing and cross checked it just now. Everything is fine.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 10, 2017, 4:26am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/4 "2017-09-10T04:26:02Z")

</div>

You can try deleting the translog files that are being referenced, but you lose whatever data is in them.

Not sure about the root cause though.

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 10, 2017, 4:27am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/5 "2017-09-10T04:27:09Z")

</div>

> [@warkolm](#):
>
> you lose whatever data is in them.

What exactly would I lose? Will it delete all the rows populated in the index?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 10, 2017, 4:33am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/6 "2017-09-10T04:33:04Z")

</div>

No, just things that may not have been written into the index. It's usually a small amount of data, but I can't say exactly what.

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 10, 2017, 4:33am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/7 "2017-09-10T04:33:53Z")

</div>

So shall I delete `indices/*/*/translog/`?

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 10, 2017, 4:37am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/8 "2017-09-10T04:37:18Z")

</div>

Is it complaining about all of them?

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 10, 2017, 5:25am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/9 "2017-09-10T05:25:55Z")

</div>

So I deleted the ones that were throwing some error. After this, there was "file not found" exception.

So I touched `.ckp` files and now there is an `EOFException` for those files -

```auto
I [2017-09-10 05:22:08,522][WARN][cluster.action.shard] [Bora] [import_profiles][4] received shard failed for [import_profiles][4], node[LYLghrxsSMOBydgtD_Xzgg], [P], v[141], s[INITIALIZING], a[id=i0RPT1IhT3yGgPoRKF7mUQ], unassigned_info[[reason=ALLOCATION_FAILED], at[2017-09-10T05:22:08.495Z], details[failed recovery, failure IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]], indexUUID [mMC5fBsDS9yVan4DpKp5ng], message [failed recovery], failure [IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException; ]
 
I [import_profiles][[import_profiles][4]] IndexShardRecoveryException[failed to recovery from gateway]; nested: EngineCreationFailureException[failed to create engine]; nested: EOFException;
 
I at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:258)
 
I at org.elasticsearch.index.shard.StoreRecoveryService.access$100(StoreRecoveryService.java:60)
 
I at org.elasticsearch.index.shard.StoreRecoveryService$1.run(StoreRecoveryService.java:133)
 
I at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 
I at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 
I at java.lang.Thread.run(Thread.java:745)
 
I Caused by: [import_profiles][[import_profiles][4]] EngineCreationFailureException[failed to create engine]; nested: EOFException;
 
I at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:135)
 
I at org.elasticsearch.index.engine.InternalEngineFactory.newReadWriteEngine(InternalEngineFactory.java:25)
 
I at org.elasticsearch.index.shard.IndexShard.newEngine(IndexShard.java:1349)
 
I at org.elasticsearch.index.shard.IndexShard.createNewEngine(IndexShard.java:1344)
 
I at org.elasticsearch.index.shard.IndexShard.internalPerformTranslogRecovery(IndexShard.java:889)
 
I at org.elasticsearch.index.shard.IndexShard.performTranslogRecovery(IndexShard.java:866)
 
I at org.elasticsearch.index.shard.StoreRecoveryService.recoverFromStore(StoreRecoveryService.java:249)
 
I ... 5 more
 
I Caused by: java.io.EOFException
	at org.apache.lucene.store.InputStreamDataInput.readByte(InputStreamDataInput.java:37)
	at org.apache.lucene.store.DataInput.readInt(DataInput.java:101)
	at org.apache.lucene.store.DataInput.readLong(DataInput.java:157)
	at org.elasticsearch.index.translog.Checkpoint.<init>(Checkpoint.java:53)
	at org.elasticsearch.index.translog.Checkpoint.read(Checkpoint.java:82)
	at org.elasticsearch.index.translog.Translog.<init>(Translog.java:165)
	at org.elasticsearch.index.engine.InternalEngine.openTranslog(InternalEngine.java:188)
	at org.elasticsearch.index.engine.InternalEngine.<init>(InternalEngine.java:131)
	... 11 more

```

Should I have not deleted the `ckp` files? How can I recover the index from here now?

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 20, 2017, 9:25am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/10 "2017-09-20T09:25:41Z")

</div>

@warkolm: Could you please take a look here again? Thanks in advance.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 20, 2017, 9:41am UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/11 "2017-09-20T09:41:02Z")

</div>

I really don't know sorry, this is not anything I have seen or know how to recover from.

Did you take backups?

---

<div class="post-metadata">

### Author: ![singhpratyush](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/singhpratyush/32/28394_2.png) [@singhpratyush](https://discuss.elastic.co/u/singhpratyush)
#### Post date: [September 25, 2017, 4:48pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/12 "2017-09-25T16:48:58Z")

</div>

> [@warkolm](#):
>
> Did you take backups?

I'm afraid no. Learning it the hard way.

Thanks for sticking around and helping out 😄

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [September 25, 2017, 10:37pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/13 "2017-09-25T22:37:59Z")

</div>

Would [https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html#corrupt-translog-truncation](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules-translog.html#corrupt-translog-truncation) help at all?

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 23, 2017, 10:38pm UTC](https://discuss.elastic.co/t/shard-recovery-fails-after-resizing-google-cloud-platforms-persistent-disk/99936/14 "2017-10-23T22:38:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.