Nfs filesystem: DirectoryIteratorException No error information

I am running a 3 node elasticsearch cluster with the following configuration
Version: 6.5.4
Environment: Docker
Xmx = 3.75
Filesystem type: nfs
Replication: 1

I am getting following error in stdout logs. [pastebin], adding few lines below for quick readability.
The error started coming on all 3 nodes, Size of data directory being 22.8G, 29.8G, 22.4G on the 3 nodes respectively.

[2019-08-17T01:25:03,582][WARN ][o.e.i.c.IndicesClusterStateService] [spotlight-es-0-athena-18] [[audits][2]] marking and sending shard failed due to [shard failure, reason [corrupt [11/1106]
Caused by: org.apache.lucene.index.CorruptIndexException: Hit unexpected exception while reading segment infos (resource=commit(null))
Caused by: java.nio.file.DirectoryIteratorException: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/indices/J0ltYZOfSs-SW5N7gPoo1Q/2/index: No error information
Caused by: java.nio.file.FileSystemException: /usr/share/elasticsearch/data/nodes/0/indices/J0ltYZOfSs-SW5N7gPoo1Q/2/index: No error information

How do I debug this further? Can it be because of nfs filesystem used as index storage?

Yes. Elasticsearch does not supports NFS filesystem.

Why is this a WARN and not an error? All the 3 nodes are printing the same stacktrace.

Probably because it is the replica that is failing, not the primary shard.

1 Like

I see that makes sense. Thanks a lot Thiago.

Elasticsearch doesn't treat any shard failure as an error whether it's the primary or a replica, because there will normally be another copy of the shard elsewhere to take over. It's something you might want to investigate, but doesn't need immediate action, so a WARN makes more sense than an ERROR.

2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.