# CorruptIndexException missing .si file

**URL:** https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994
**Category:** Elasticsearch
**Created:** [June 13, 2020, 6:59pm UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994 "2020-06-13T18:59:55Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![keduadoi](https://avatars.discourse-cdn.com/v4/letter/k/4af34b/32.png) [@keduadoi](https://discuss.elastic.co/u/keduadoi)
#### Post date: [June 13, 2020, 6:59pm UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/1 "2020-06-13T18:59:55Z")

</div>

Our cluster has 3 nodes, primary shards are about 10K, and replica = 1.  
The cluster had worked normally until 1 cluster showed error message:

```auto
[root@server2014 ~]# uncaught exception in thread [main]
ElasticsearchException[failed to bind service]; nested: CorruptIndexException[Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/home/elasticsearch/polaris/nodes/0/_state/segments_3on4")))]; nested: NoSuchFileException[/home/elasticsearch/polaris/nodes/0/_state/_33n9.si];
Likely root cause: java.nio.file.NoSuchFileException: /home/elasticsearch/polaris/nodes/0/_state/_33n9.si
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)

```

That node cannot started again, the error message always appeared, the missing file actually could not be found, we don't have any snapshot so we lost a part of our data.

So I would like to know what could be the root cause for this situation? Our elasticsearch version is 7.6. And how can we prevent this problem in the future? I'm thinking about increasing replicate and use snapshot. Thank you in advance.

---

<div class="post-metadata">

### Author: ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 13, 2020, 7:15pm UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/2 "2020-06-13T19:15:54Z")

</div>

It sounds like [you have far too many shards in your cluster](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). It may not be directly related to your error message, but is likely to be causing problems.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 14, 2020, 8:02am UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/3 "2020-06-14T08:02:12Z")

</div>

> [@keduadoi](#):
>
> So I would like to know what could be the root cause for this situation?

Either the file was deleted by an external force after Elasticsearch wrote it, or else your storage is misconfigured to ignore fsync calls and there was a power outage.

---

<div class="post-metadata">

### Author: ![keduadoi](https://avatars.discourse-cdn.com/v4/letter/k/4af34b/32.png) [@keduadoi](https://discuss.elastic.co/u/keduadoi)
#### Post date: [June 14, 2020, 9:03am UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/4 "2020-06-14T09:03:02Z")

</div>

Can you make it a little bit more clarification? If power outage was the case, what would happened to delete the \*.si file, then why we need and how can we "configure to ignore fsync calls"? I don't have much time to work with elasticsearch so I may ask too basic questions, I'm very appreciated for any instruction.

---

<div class="post-metadata">

### Author: ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 14, 2020, 10:20am UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/5 "2020-06-14T10:20:44Z")

</div>

> [@keduadoi](#):
>
> what would happened to delete the \*.si file

In this case, it's less that the file was deleted and more that it was never actually written in the first place. These things happened in order:

1. Elasticsearch wrote `nodes/0/_state/_33n9.si` and called `fsync()` to ensure that this write was durable (i.e. that it will persist across a power outage).
2. The disk acknowledged the `fsync()` to confirm that the write was indeed durable.
3. Elasticsearch wrote `nodes/0/_state/segments_3on4` (which refers to `_33n9.si`).
4. The disk completes a durable write of `nodes/0/_state/segments_3on4`.

Step 2 is where this often falls down if your system is misconfigured: the disk claims to have durably written the file without actually having done so. This is often the default behaviour since durable writes can be slow and you get better performance numbers by lying like this. If the write wasn't really durable and then there's a power outage then when the node restarts it finds that the file simply isn't there: it was never really written.

This isn't really anything to do with Elasticsearch - it has to assume that you have configured everything for durable writes, and if that's not the case then you'll lose data.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 12, 2020, 10:20am UTC](https://discuss.elastic.co/t/corruptindexexception-missing-si-file/236994/6 "2020-07-12T10:20:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
