# Checksum failed (hardware problem?)
**URL:** https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061
**Category:** Elasticsearch
**Created:** [August 17, 2023, 9:13pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061 "2023-08-17T21:13:11Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@bigjohns97](https://discuss.elastic.co/u/bigjohns97)
#### Post date: [August 17, 2023, 9:13pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/1 "2023-08-17T21:13:11Z")
I am using a simple one node Elasticsearch instance on a Debian VM running on top of a Hyper-V Windows 11 system with an AMD 3950 CPU DDR4 RAM and SSD disk. Using elasticsearch as a index for a graylog instance and sending logs from my network equipment to this via syslog I am setting retention via Graylog to cycle every 7 days.
Everytime this cycles I notice this error about checksum failed during the merge and creation of the new index.
I have to manually go in and delete the old index for the red alert to clear in graylog and when tracking down the error through elasticsearch I come up with this checksum failed error.
Here is the full output
```auto
curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
{
"index": "pfsense_filterlog_28",
"shard": 1,
"primary": true
}
'
{
"index" : "pfsense_filterlog_28",
"shard" : 1,
"primary" : true,
"current_state" : "unassigned",
"unassigned_info" : {
"reason" : "ALLOCATION_FAILED",
"at" : "2023-08-13T01:52:26.419Z",
"failed_allocation_attempts" : 1,
"details" : "failed shard on node [ktrLh1vHTAuk1iKoGbEuTQ]: shard failure, reason [merge failed], failure MergeException[org.apache.lucene.index.CorruptIndexException: checksum failed (hardware problem?) : expected=3a7148ca actual=b0e6e03 (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/var/lib/elasticsearch/nodes/0/indices/fS9fQIj0RjioMdvD0t4BuQ/1/index/_48yy.cfs\") [slice=_48yy_Lucene84_0.tim]))]; nested: CorruptIndexException[checksum failed (hardware problem?) : expected=3a7148ca actual=b0e6e03 (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/var/lib/elasticsearch/nodes/0/indices/fS9fQIj0RjioMdvD0t4BuQ/1/index/_48yy.cfs\") [slice=_48yy_Lucene84_0.tim]))]; ",
"last_allocation_status" : "no_valid_shard_copy"
},
"can_allocate" : "no_valid_shard_copy",
"allocate_explanation" : "cannot allocate because all found copies of the shard are either stale or corrupt",
"node_allocation_decisions" : [
{
"node_id" : "ktrLh1vHTAuk1iKoGbEuTQ",
"node_name" : "debian",
"transport_address" : "127.0.0.1:9300",
"node_decision" : "no",
"store" : {
"in_sync" : true,
"allocation_id" : "B-NovUobT_68k3UeF2gcQQ",
"store_exception" : {
"type" : "corrupt_index_exception",
"reason" : "failed engine (reason: [merge failed]) (resource=preexisting_corruption)",
"caused_by" : {
"type" : "i_o_exception",
"reason" : "failed engine (reason: [merge failed])",
"caused_by" : {
"type" : "corrupt_index_exception",
"reason" : "checksum failed (hardware problem?) : expected=3a7148ca actual=b0e6e03 (resource=BufferedChecksumIndexInput(MMapIndexInput(path=\"/var/lib/elasticsearch/nodes/0/indices/fS9fQIj0RjioMdvD0t4BuQ/1/index/_48yy.cfs\") [slice=_48yy_Lucene84_0.tim]))"
}
}
}
}
}
```
I know people usually point to hardware errors here but I don't have any hardware errors, I have plenty of other indices running on here that never have this issue and no other issues outside of Elasticsearch.
I have also checked the filesystem on boot and pre-mount and even set the OS to do this every time the system is booted and no issues with the FS is found.
Any idea what is happening here?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [August 18, 2023, 6:07am UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/2 "2023-08-18T06:07:39Z")
There's some information about this in the manual:
> **[Troubleshooting corruption | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/corruption-troubleshooting.html#corruption-troubleshooting)**
TLDR the exception means that the data Elasticsearch read from storage is not the data that it originally wrote there. ES has no practical way of determining why it might be so.
> [@bigjohns97](#):
>
> I don't have any hardware errors [...] no issues with the FS is found
Note the following paragraph from the docs I linked:
> Data corruption typically doesn’t result in other evidence of problems apart from the checksum mismatch. Do not interpret this as an indication that your storage subsystem is working correctly and therefore that Elasticsearch itself caused the corruption. It is rare for faulty storage to show any evidence of problems apart from the data corruption, but data corruption itself is a very strong indicator that your storage subsystem is not working correctly.
---
### Author:  [@bigjohns97](https://discuss.elastic.co/u/bigjohns97)
#### Post date: [August 19, 2023, 3:03pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/3 "2023-08-19T15:03:06Z")
> Data corruption typically doesn’t result in other evidence of problems apart from the checksum mismatch. Do not interpret this as an indication that your storage subsystem is working correctly and therefore that Elasticsearch itself caused the corruption. It is rare for faulty storage to show any evidence of problems apart from the data corruption, but data corruption itself is a very strong indicator that your storage subsystem is not working correctly.
How come none of my other rotating indices have issues?
How come I can't find any evidence of data corruption anywhere else on the system?
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [August 19, 2023, 6:40pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/4 "2023-08-19T18:40:44Z")
These are good questions, but the simplest way to answer them is covered in the docs I linked:
> To narrow down the source of the corruptions, systematically change components in your cluster’s environment until the corruptions stop.
Data corruption bugs often need a very particular access pattern to trigger. For instance the [one we blogged about a few years back](https://www.elastic.co/blog/canonical-elastic-and-google-team-up-to-prevent-data-corruption-in-linux) was only seen after several days of indexing at full speed.
---
### Author:  [@bigjohns97](https://discuss.elastic.co/u/bigjohns97)
#### Post date: [August 19, 2023, 8:16pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/5 "2023-08-19T20:16:32Z")
I guess I will start with a new index and go from there. Will definitely post back whenever I find out what it is
---
### Author:  [@bigjohns97](https://discuss.elastic.co/u/bigjohns97)
#### Post date: [September 2, 2023, 12:28pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/6 "2023-09-02T12:28:11Z")
This ended up being bad RAM, discovered via bootable memtest86.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [September 2, 2023, 2:07pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/7 "2023-09-02T14:07:09Z")
Thanks for following up here @bigjohns97, much appreciated! Bad RAM would definitely explain what you were seeing.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 30, 2023, 2:07pm UTC](https://discuss.elastic.co/t/checksum-failed-hardware-problem/341061/8 "2023-09-30T14:07:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.