I am running a 3 node elasticsearch cluster with the following configuration
Version: 6.5.4
Environment: Docker
Xmx = 3.75
Filesystem type: nfs
Replication: 1
I am getting following error in stdout logs. [pastebin], adding few lines below for quick readability.
The error started coming on all 3 nodes, Size of data directory being 22.8G, 29.8G, 22.4G on the 3 nodes respectively.
[2019-08-17T01:25:03,582][WARN ][o.e.i.c.IndicesClusterStateService] [spotlight-es-0-athena-18] [[audits][2]] marking and sending shard failed due to [shard failure, reason [corrupt [11/1106]
Caused by: org.apache.lucene.index.CorruptIndexException: Hit unexpected exception while reading segment infos (resource=commit(null))
Caused by: java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/indices/J0ltYZOfSs-SW5N7gPoo1Q/2/index: No error information
Caused by: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/indices/J0ltYZOfSs-SW5N7gPoo1Q/2/index: No error information
How do I debug this further? Can it be because of nfs filesystem used as index storage?