Issue using a sshfs docker mount as a snapshot repository

Elastic Stack Elasticsearch
,
1

Hello,

I'm having trouble trying to get Elasticsearch to register a docker mount as a snapshot repository.

The situation

I'm migrating indices from a server to another, let's call them "old host" and "new host". Both use Docker Swarm and are running an Elasticsearch container based on the same image (7.17).
A snapshot has been successfully created on the old host and I'm struggling really hard to get the new server to register the old host repository onto the new one in order to restore the snapshot on the new host.
I tried almost everything and currently am transferring the snapshot files using scp to try to get the new host to somehow restore from those files as a last resort attempt.

What I tried

  1. Opening an SSH tunnel to expose the old host Elasticsearch and register an URL snapshot repository

I used the tunnel for reindexing: it works but it's too slow to move every indices from the old host to the new. Point is I can access ES on the old host, I can even register a repository based on this URL (after having it added to the repositories.url.allowed_urls) but it does not list the snapshot created on the old host when using GET /_cat/snapshots/<repository> and, of course, it does not work either when I try to restore the snapshot.
I used "url": "http://172.17.0.1:9201" when registering the remote snapshot repository, should I have added a trailing slash? A specific path?
Sorry to insist but, just in case: the URL provided above is accessible from the ES container on the new host, so I know it works.

  1. Mounting the remote snapshot directory using SSHFS and register an FS snapshot repository

This one is quite straightforward: I mounted the remote snapshot directory using SSHFS, mounted this directory onto the ES container on the new host, checked I can access the files, checked the access rights (all good, 1000:0 everywhere, directory mounted on / so no path-traversal problem) and tried to register the fs repository using the "location": "/old_host_es_backup" as settings.
But first, you have to declare this path into the path.repo setting and that's where things get interesting because if I do that, Elasticsearch will not start and throw those errors:

"stacktrace": ["org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/old_host_es_backup)",
[…]
"Caused by: java.nio.file.NotDirectoryException: /old_host_es_backup",
"at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:321) ~[elasticsearch-7.17.1.jar:7.17.1]",
uncaught exception in thread [main]
[…]
"Suppressed: java.nio.file.FileAlreadyExistsException: /old_host_es_backup",
[…]
Likely root cause: java.nio.file.NotDirectoryException: /old_host_es_backup

Again: I checked the directory is accessible from the container and its access rights, everything is fine. Should I check it's writeable? Can I add a file to this directory as a test without disturbing the old host ES?

  1. Mounting the remote snapshot directory using SSHFS and register as an URL snapshot repository

As a mix between the two previous attempts, I tried to register the repository as a read-only URL with the "url": "file:/old_host_es_backup" setting:

curl \
    -X PUT "localhost:9200/_snapshot/old_host_fs_repository?pretty" \
    -H 'Content-Type: application/json' \
    -d'
{
  "type": "url",
  "settings": {
    "url": "file:/old_host_es_backup"
  }
}
'

but it resulted in the following error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[old_host_fs_repository] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[old_host_fs_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "access_control_exception",
      "reason" : "access denied (\"java.io.FilePermission\" \"/index.latest\" \"read\")"
    }
  },
  "status" : 500
}

I know the documentation state that "URLs using the file protocol must point to the location of a shared filesystem accessible to all master and data nodes in the cluster. This location must be registered in the path.repo setting." and that it may be the problem but I had to try and as you saw earlier I cannot add the location to the path.repo setting.

Any help would be greatly appreciated as I'm running out of ideas to get this working.

2

I'm kinda disappointed that no one responded (not even the support), but in case someone find this post later on:
I have been able to restore a snapshot by copying the repository files from a server to another, which is somewhat implied in Back up a repository (and in a few tech posts around the internet) and then registering the destination folder as an fs repository.

Let it be known: the documentation is really lacking examples and can be quite vague about how to transfer data from a server to another.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.