Crash using elasticsearch-shard tool to recover corrupt index (EOF)

craigsheppard · July 5, 2019, 6:53pm

Had a power outage that took down a small (one node, 2 shards, two replicas) locally hosted elastic stack (was using sebp/elk 7.1 at the time, since upgraded to 7.2).

Bringing the stack back online resulted in Kibana complaining that it couldn't load any data from my index. Digging around, I found that both of my primaries were in the UNASSIGNED ALLOCATION_FAILED state.

I used the reroute API with allocate_stale_primary but was unsuccessful.

I tried running the elasticsearch-shard tool inside the container, but it gave me a nullPointerException, so on the advice of Dimitrios Liappis from the elastic repo, I tried with the official elastic docker image.

This time, I get the following output:

[elasticsearch@51a35514f16e ~]$ bin/elasticsearch-shard remove-corrupted-data --index environment --shard-id 1   
ERROR StatusLogger No Log4j 2 configuration file found. Using default configuration (logging only errors to the console), or user programmatically provided configurations. Set system property 'log4j2.debug' to show Log4j 2 internal initialization logging. See https://logging.apache.org/log4j/2.x/manual/configuration.html for instructions on how to configure Log4j 2
-----------------------------------------------------------------------

    WARNING: Elasticsearch MUST be stopped before running this tool.

  Please make a complete backup of your index before using this tool.

-----------------------------------------------------------------------

Opening Lucene index at /usr/share/elasticsearch/data/nodes/0/indices/8WitgzPDQvCiYJcnnli1sQ/1/index


 >> Lucene index is clean at /usr/share/elasticsearch/data/nodes/0/indices/8WitgzPDQvCiYJcnnli1sQ/1/index


Opening translog at /usr/share/elasticsearch/data/nodes/0/indices/8WitgzPDQvCiYJcnnli1sQ/1/translog

read past EOF. pos [51026] length: [4] end: [51026]
Exception in thread "main" java.io.EOFException: read past EOF. pos [51026] length: [4] end: [51026]
	at org.elasticsearch.common.io.Channels.readFromFileChannelWithEofException(Channels.java:103)
	at org.elasticsearch.index.translog.TranslogSnapshot.readBytes(TranslogSnapshot.java:104)
	at org.elasticsearch.index.translog.BaseTranslogReader.readSize(BaseTranslogReader.java:79)
	at org.elasticsearch.index.translog.TranslogSnapshot.readOperation(TranslogSnapshot.java:80)
	at org.elasticsearch.index.translog.TranslogSnapshot.next(TranslogSnapshot.java:70)
	at org.elasticsearch.index.translog.MultiSnapshot.next(MultiSnapshot.java:69)
	at org.elasticsearch.index.translog.TruncateTranslogAction.isTranslogClean(TruncateTranslogAction.java:187)
	at org.elasticsearch.index.translog.TruncateTranslogAction.getCleanStatus(TruncateTranslogAction.java:86)
	at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.lambda$execute$1(RemoveCorruptedShardDataCommand.java:329)
	at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.findAndProcessShardPath(RemoveCorruptedShardDataCommand.java:202)
	at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.execute(RemoveCorruptedShardDataCommand.java:282)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:77)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.index.shard.ShardToolCli.main(ShardToolCli.java:35)

I'm not overly concerned with losing some incremental data, but I would like to recover the 12 months of data already stored in the cluster, AND begin to capture data again.

Unfortunately, my snapshots were misconfigured, and the last one was taken the first time I restarted the cluster (about a year ago ).

So... Does anyone have a suggestion as to how I can get back up and running again?

Thanks so much!

Craig

DavidTurner · July 5, 2019, 7:25pm

Unfortunately this version of elasticsearch-shard doesn't correctly handle truncated translog files. This is fixed in #42744 but that's not released yet. The safest thing to do is to wait for the release of 7.3, but I can't tell you when that'll be. If you feel the need to try any other course of action, make sure that you can get the node back into exactly the state that it's in now, otherwise there may be no hope of recovery in the future.

If you had one node, in practice you will have had no replicas. Elasticsearch only allocates a single copy of each shard (either primary or replica) to each node, so if there's only one node it will only allocate the primaries.

Also this failure indicates that your storage system is not working correctly. A power outage alone doesn't cause this kind of corruption.

craigsheppard · July 5, 2019, 7:58pm

Thanks, David! I do have a filesystem snapshot of the state at the time of the coutage.

Perhaps the corruption was due to a lack of disk space - from my startup logs:

[2019-07-05T19:54:11,841][WARN ][o.e.c.r.a.DiskThresholdMonitor] [elk] high disk watermark [90%] exceeded on [VfKITtDXQoOV_k4HtBsm3g][elk][/var/lib/elasticsearch/nodes/0] free: 59.4gb[6.3%], shards will be relocated away from this node
[2019-07-05T19:54:11,841][INFO ][o.e.c.r.a.DiskThresholdMonitor] [elk] rerouting shards: [high disk watermark exceeded on one or more nodes]

Either way - I just want to get back up and running with as much of the data as possible - they are not mission critical - just historical environmental sensor data from a bunch of devices installed around my home. I think in the meantime I'll create a new cluster and start logging data to a new location, and try to export/import the historical data when 7.3 comes out.

Thanks for your help!

DavidTurner · July 5, 2019, 8:12pm

If you are confident in this snapshot then you could try the following. Look for a file within /usr/share/elasticsearch/data/nodes/0/indices/8WitgzPDQvCiYJcnnli1sQ/1/translog of size 51026 bytes (maybe not exactly, but should be very close). I expect it will have a name translog-NNN.tlog. Try moving it elsewhere (out of /usr/share/elasticsearch) and then try elasticsearch-shard again. If that doesn't work, put it back how it was

No, it's not that. 6.3% is loads (!) of space. This corruption indicates that some writes were reported as successful by the OS before they had actually hit the disk, and then the writes that ultimately hit the disk did so in an invalid order.

craigsheppard · July 11, 2019, 2:02am

Thanks David - I finally had some time to attempt this - I found the 51kb file (all the rest were either 55b or 88b), but now when I run the Elasticsearch-shards utility I get the error:

translog file doesn't exist with generation: 195 recovering from: 195 checkpoint: 241 - translog ids must be consecutive

I guess I'll hang tight until 7.3.

DavidTurner · July 11, 2019, 11:12am

Darn, you're quite right, we aren't handling that case properly. I opened https://github.com/elastic/elasticsearch/pull/44217 to address this. Thanks for the feedback.

system · August 8, 2019, 11:12am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.