Input/output error when creating snapshot

Elastic Stack Elasticsearch
1

Hi!

We have a 3-nodes self-hosted ES 8.5 cluster, and when we make daily snapshots few shards fail every time with the following error:

INTERNAL_SERVER_ERROR: UncategorizedExecutionException[Failed execution]; nested: ExecutionException[java.io.IOException: Input/output error]; nested: IOException[Input/output error]

Snapshots are stored on a standalone server, mounted as NFS on all ES nodes.

This is how snapshot repository is configured:

curl -k -u "elastic:password" -XPUT 'https://10.1.2.3:9200/_snapshot/nyx?pretty' -H 'Content-Type: application/json' -d '{
"type": "fs",
"settings": {
"location": "/mnt/backup/snapshots",
"compress": true
}
}'

Can you point me towards the way to debug these errors please?

2

Input/output error is a low-level error message reported by the OS itself - see the entry for EIO on this page:

Typically that means your hardware is broken, or something else is fundamentally wrong in your system, but this is nothing to do with Elasticsearch.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.