Snaphsot repository Read Only URL via FTP

Elastic Stack Elasticsearch
1

I have an Elasticsearch cluster of 3 nodes with version 7.15.0.
I want to add a read-only URL repository soi can fetch a snapshot for restore.

PUT /_snapshot/ftp_snaphoot
{
  "type": "url",
  "settings": {
    "url": "ftp://username:password@ftp_server_ip_adress"
  }
}

I got this error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[ftp_snaphoot] Could not determine repository generation from root blobs"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[ftp_snaphoot] Could not determine repository generation from root blobs",
    "caused_by" : {
      "type" : "security_exception",
      "reason" : "access denied (\"java.net.SocketPermission\" \"localhost:0\" \"listen,resolve\")"
    }
  },
  "status" : 500
}

When the username login to ftp_server_ip_adress using FTP, it's directly located into a respository with a snapshot. This respository is used as FS to take snapshot from the cluster.

2

This is the stack trace reported by in Elasticsearch logs

{"type": "server", "timestamp": "2021-10-23T16:33:46,248Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "docker-cluster", "node.name": "elasticsearch_1", "message": "path: /_snapshot/ftp_snaphoot/_verify, params: {repository=ftp_snaphoot}", "cluster.uuid": "JLG6niOySXmX9mJkzv1I0w", "node.id": "D0H2EJblRmaBrwohfQBFPw" ,
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [elasticsearch_3][172.18.0.3:9300][cluster:admin/repository/verify]",
"Caused by: org.elasticsearch.repositories.RepositoryVerificationException: [ftp_snaphoot] path  is not accessible on master node",
"Caused by: java.lang.SecurityException: access denied (\"java.net.SocketPermission\" \"localhost:0\" \"listen,resolve\")",
"at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]",
"at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]",
"at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]",
"at java.lang.SecurityManager.checkListen(SecurityManager.java:990) ~[?:?]",
"at java.net.Socket.bind(Socket.java:691) ~[?:?]",
"at sun.net.ftp.impl.FtpClient.openPassiveDataConnection(FtpClient.java:691) ~[?:?]",
"at sun.net.ftp.impl.FtpClient.openDataConnection(FtpClient.java:806) ~[?:?]",
"at sun.net.ftp.impl.FtpClient.getFileStream(FtpClient.java:1384) ~[?:?]",
"at sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:460) ~[?:?]",
"at java.net.URL.openStream(URL.java:1192) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:554) ~[?:?]",
"at org.elasticsearch.common.blobstore.url.URLBlobContainer.getInputStream(URLBlobContainer.java:142) ~[?:?]",
"at org.elasticsearch.common.blobstore.url.URLBlobContainer.readBlob(URLBlobContainer.java:110) ~[?:?]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.readSnapshotIndexLatestBlob(BlobStoreRepository.java:2563) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.latestIndexBlobId(BlobStoreRepository.java:2554) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:1655) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at org.elasticsearch.repositories.RepositoriesService$4.doRun(RepositoriesService.java:392) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:737) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.15.0.jar:7.15.0]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) ~[?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }
3

It looks like FTP-based URL repositories simply don't work: the FTP client is trying to do something that the security manager forbids. This code hasn't changed in well over a decade so I guess you're the first person to actually try doing this. Please report it on Github as a bug - at the very least we should remove ftp from the list of URL schemes that are documented as supported. In the meantime I'd suggest using HTTP instead.

4

Thanks @DavidTurner for your feedbakc
I reported an issue here Snaphsot repository Read Only URL via FTP not accepting FTP URL · Issue #79685 · elastic/elasticsearch · GitHub
Hope that will help.

My idea is to use FTP repo on a remote cluster when i need to restore the snaphot from the main cluster. and using an FTP repo (ReadOnly) is a good option for me to avoid mounting NFS ...

1 Like
5

Yes, makes sense, but FTP is a fairly obscure protocol these days as you have discovered. It doesn't do anything that HTTP(S) can't.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.