Index got corrupted without any crash

Elastic Stack Elasticsearch
1

Hi,

We have one single node elastic cluster which was running fine till last week. From last week we saw frequently one index got corrupted and went in 'RED' state without any system crash and disk space problem.

When I see a particular index folder I found one corrupted_* file was present with following error

?×lstore Ð
&failed engine (reason: [merge failed])Gchecksum failed (hardware problem?) : expected=8544add4 actual=77fd1c54©BufferedChecksumIndexInput(MMapIndexInput(path="C:\Program Files\Klera\elasticsearch\data\nodes\0\indices\pVdJlhg2SMSA9sjWUtROMA\0\index_38mus.cfs") [slice=_38mus.fdt])

Our team also diagnose hardware and found everything is fine.

Please suggest how to resolve this problem.

Thanks in advance.

2

Which version of Elasticsearch are you using? Which operating system? What kind of storage are you using?

3

Hi,

Please find details of system as follow.

Elasticsearch :- 7.1.1
OS :- Window server 2019 standard
Storage type :- file system (SSD drive).
RAM :- 32 GB

We are using single node elastic cluster with configuration
-Xms4g
-Xmx4g
-XX:MaxDirectMemorySize=1g

4

I recall seeing some report related to corruption in some Linux kernels but have no experience with running Elasticsearch on Windows.

5

Thanks @Christian_Dahlqvist.

What can be a possible cause of such a type of corruption? Can you give me some direction so that I can check and identify the reasons for this?

6

I do not know. I would recommend deleting the corrupt index and restoring it from a snapshot.

7

Actually when I removed that corrupted_* file from folder and start the elasticsearch index recovered and work fine for some time.

Can you suggest someone who can help me in this problem.

Thanks

8

Don't ever delete or otherwise alter the contents of the data directory yourself.

This index was found to be corrupt, almost certainly due to a hardware issue as the message suggests. Disks are nowhere near 100% reliable so this will happen from time to time, and with a single node there's no way for Elasticsearch to recover your data itself. The index may well still be corrupt but Elasticsearch is ignoring it for now because you fiddled with the contents of the data path yourself.

1 Like
9

Thanks @DavidTurner

I will take care of it.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.