# Version 7.1.1 Corrupted translog After a power failure

**URL:** https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968
**Category:** Elasticsearch
**Created:** [October 31, 2019, 3:50am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968 "2019-10-31T03:50:24Z")
**Posts on this page:** 14
**Page:** 1

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 3:50am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/1 "2019-10-31T03:50:24Z")

</div>

Translog has become corrupted and why would this happen. Can close system file cache fix the Problem Completely?

"unassigned\_info": {  
"reason": "ALLOCATION\_FAILED",  
"at": "2019-10-09T07:08:10.690Z",  
"failed\_attempts": 5,  
"delayed": false,  
"details": "failed shard on node [aTbLQpDwTL204ASeorrvsA]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: EOFException[read past EOF. pos [1070861] length: [4] end: [1070861]]; ",  
"allocation\_status": "deciders\_no"  
}

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 4:00am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/2 "2019-10-31T04:00:12Z")

</div>

The problem comes almost every time after power off system while inserting data into elasticsearch  
But after I try to close system file cache, the problem seems to disappear. Does't it really work? Who can analysis principly?

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 31, 2019, 5:50am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/3 "2019-10-31T05:50:23Z")

</div>

> [@panxuelin](#):
>
> Translog has become corrupted and why would this happen.

The usual explanation is that your storage is not working correctly and is acknowledging writes before they have completed. This is a trick that lower-grade storage sometimes uses to improve its performance numbers at the expense of your data.

> [@panxuelin](#):
>
> But after I try to close system file cache, the problem seems to disappear.

What do you mean "close system file cache"?

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 6:31am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/4 "2019-10-31T06:31:53Z")

</div>

Yes, "close system file cache" by using linux command "hparm -W"

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 6:37am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/5 "2019-10-31T06:37:06Z")

</div>

Do you mean It`s problem of storage Or if something wrong in "writing translog"?  
If "close system file cache" can solve the problem.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 31, 2019, 6:37am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/6 "2019-10-31T06:37:48Z")

</div>

What type of storage are you using? Is it some kind of network attached filesystem?

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 6:39am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/7 "2019-10-31T06:39:10Z")

</div>

What kind of parameter you mean?

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 31, 2019, 6:39am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/8 "2019-10-31T06:39:49Z")

</div>

What type of hardware is the cluster deployed on?

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 6:41am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/9 "2019-10-31T06:41:05Z")

</div>

We try HDD and SSD. Same problem.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 31, 2019, 6:43am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/10 "2019-10-31T06:43:54Z")

</div>

> [@panxuelin](#):
>
> hparm -W

This disables the write cache and does indeed indicate that your disk is lying to Elasticsearch and acknowledging writes before they have completed.

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 6:52am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/11 "2019-10-31T06:52:07Z")

</div>

If It means that hparm can solve the problem.Are there something else cause corrupted translog.  
where can I find flow path of writing translog.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [October 31, 2019, 7:06am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/12 "2019-10-31T07:06:32Z")

</div>

No, this does not indicate a problem in Elasticsearch or elsewhere. It indicates that your disks have a volatile write cache that loses data on a power loss.

---

<div class="post-metadata">

### Author: ![panxuelin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/panxuelin/32/56743_2.png) [@panxuelin](https://discuss.elastic.co/u/panxuelin)
#### Post date: [October 31, 2019, 7:10am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/13 "2019-10-31T07:10:11Z")

</div>

Thank you !

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 28, 2019, 7:10am UTC](https://discuss.elastic.co/t/version-7-1-1-corrupted-translog-after-a-power-failure/205968/14 "2019-11-28T07:10:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
