Hello,
I have confirmed a valid path in my path.repo setting, but now when I do a 'verify repository' it gives me an error:
{
"name": "ResponseError",
"message": "repository_verification_exception\n\tCaused by:\n\t\tdirectory_not_empty_exception: /mnt/elastic-share/es-backups/tests-QVpHYVK-S26hF3FqOI6dxg\n\tRoot causes:\n\t\trepository_verification_exception: [Before-818upgrade-6_19_2025] cannot delete test data at "
}
And the verification status remains 'Not Connected'
The 'tests-<anything>' continues to appear in there without getting deleted. I checked permissions but it's wide open for elasticsearch user. I also confirmed that after deleting the tests item that appears in there, if I click the verify repository again, it fails with the same error and a new 'tests-<anything>' has appeared.
We're working on performing an upgrade to 9.x, so have to upgrade to 8.18 first, which is where this started appearing.
This means your repository at /mnt/elastic-share/es-backups/ is not behaving like a proper filesystem (at least, not one with Elasticsearch-exclusive access). Elasticsearch created a handful of files in /mnt/elastic-share/es-backups/tests-QVpHYVK-S26hF3FqOI6dxg/ and then deleted them, but either the deletion did not work or something else also created a file in this directory at the same time. Either way, the directory wasn't empty afterwards so couldn't be deleted, which indicates that this filesystem is not safe to use as a snapshot repository.
Thanks @DavidTurner -- I managed to get the backup configured using the top folder /mnt/elastic-share as the location. This worked, but I am curious if we have the backup folder configured incorrectly?
I didn't see any documentation describing who the owner of that repo should be or the permissions it should have. Do you have that information handy?
Currently it's configured as
o/g: Root perm: 644
No special configuration required here. ES isn't doing any magic, it's just creating/reading/writing files using the standard filesystem syscalls, like any other userspace process.