Elasticsearch fails to register the repository but succeeds after retry

Elastic Stack Elasticsearch
1

The issue is only on repository on Zadara. Error:

"type": "repository_verification_exception",
"reason": "[index_test_1634127279] [[wZIWLWCURDCiCRy6B6tClw, 'RepositoryVerificationException[[index_test_1634127279] Failed to verify repository]; nested: FileSystemException[/usr/share/elasticsearch/backup/index_test_1634127279/tests-usS5N1ccQhmtO3rfYnRPKQ/master.dat: Invalid argument];']]",

Based on steps from Elasticsearch snapshot repository fails validation, it's likely failed after the temp was renamed as master.dat and can't fsync the containing directory. Not sure why retry passes the verification. Maybe just network hiccups?!

Another question is that is it possible to have repository registered even though there is repository_verification_exception? Our TechOps insists that after the exception, he can see the repo register and worry about maybe we'll get partial snpashot.

2

The invalid argument error looks more like an mmap error to me but I am not an expert in snapshot repositories.

If it's an mmap error, that often occurs when the file we are attempting to map is empty, which should not happen as the write should be atomic.

How are you mapping the snapshot repository on the system? Have you ensured you've configured the mount with proper settings to ensure read-after-write semantics?