Version 7.1.1 Corrupted translog After a power failure

Elastic Stack Elasticsearch
1

Translog has become corrupted and why would this happen. Can close system file cache fix the Problem Completely?

"unassigned_info": {
"reason": "ALLOCATION_FAILED",
"at": "2019-10-09T07:08:10.690Z",
"failed_attempts": 5,
"delayed": false,
"details": "failed shard on node [aTbLQpDwTL204ASeorrvsA]: shard failure, reason [failed to recover from translog], failure EngineException[failed to recover from translog]; nested: EOFException[read past EOF. pos [1070861] length: [4] end: [1070861]]; ",
"allocation_status": "deciders_no"
}

2

The problem comes almost every time after power off system while inserting data into elasticsearch
But after I try to close system file cache, the problem seems to disappear. Does't it really work? Who can analysis principly?

3

The usual explanation is that your storage is not working correctly and is acknowledging writes before they have completed. This is a trick that lower-grade storage sometimes uses to improve its performance numbers at the expense of your data.

What do you mean "close system file cache"?

4

Yes, "close system file cache" by using linux command "hparm -W"

5

Do you mean It`s problem of storage Or if something wrong in "writing translog"?
If "close system file cache" can solve the problem.

6

What type of storage are you using? Is it some kind of network attached filesystem?

7

What kind of parameter you mean?

8

What type of hardware is the cluster deployed on?

9

We try HDD and SSD. Same problem.

10

This disables the write cache and does indeed indicate that your disk is lying to Elasticsearch and acknowledging writes before they have completed.

1 Like
11

If It means that hparm can solve the problem.Are there something else cause corrupted translog.
where can I find flow path of writing translog.

12

No, this does not indicate a problem in Elasticsearch or elsewhere. It indicates that your disks have a volatile write cache that loses data on a power loss.

13

Thank you !

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.