Option missing "fix" on startup

One last comment. This is a very powerful tool that should probably get a bit more exposure in the documentation. I know it is a tool of last resort, but in my situation, it was able to restore the index without any data loss.

I'd recommend slipping a reference / link to the tool in a few sections of the documentation that cover recovery options.

Also, for this situation, the sequence of events was that I ran out of drive space (I turned off watermark checks and forgot to re-enable them) and in the logs, apparently a merge operation failed. It looks like ES retried and then gave up and marked the shard as corrupted (there was a file starting with corrupt in the nodes data directory for that index / shard).

My gut feeling is that ES is very liberal when it comes to marking a shard / index as corrupted (perhaps to prevent further data loss) and even though it was marked as corrupted / stale, the data itself was still preserved with no data loss when using the tool.

Hope this helps others out there.

1 Like