Elasticsearch snapshot and recovery

Elastic Stack Elasticsearch
1

Hi Team,

Am using Elasticsearch three node cluster with version 8.6.2 and hosted on a Linux environment. Doing a POC in Snapshot and Recovery and external NFS storage is configured to store the snapshots. The NFS repo is 250 GB.

  1. The snapshot was successful when I use the below command and could see the metadata files were created under the destination NFS storage.

curl -XPUT ' http://cl-elastic-prf.snapshot.com:9200/_snapshot/testnfsrepo/test-snapshot?wait_for_completion=true'

  1. Whereas, the snapshot got failed when I use the below command and could see error message related to “Failed to create blob container or Access denied exception”

curl -XPUT -k --user "elastic:xxxxxxxxxxxxxxx-" 'http://cl-elastic-prf.snapshot.com:9200/_snapshot/testnfsrepo/test-snapshot?wait_for_completion=true'

  1. This is the similar case for SLM policy, when I schedule a SLM policy without the user credentials the snapshot was successful as per the policy schedule and no error messages but when I try schedule a SLM policy with user credentials as user elastic as mentioned above but the snapshot got failed with exceptions “Failed to create blob container or Access denied exception”.

Exception:

Caused by: java.nio.file.AccessDeniedException: /etc/middleware/indices/8GVR_UIGQ6WDcBklYUpcrw
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:397) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:700) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:807) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:793) ~[?:?]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-8.6.2.jar:?]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-8.6.2.jar:?]
... 8 more
Caused by: java.nio.file.AccessDeniedException: /etc/middleware/indices/VSBpGoTdRa-uXxYyhyGj0g
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

Just wanted to know,

  1. what is the best practice to schedule the SLM policy whether scheduling them with user credentials or without user credentials.
  2. Also, whether the restoration will work fine if I schedule the SLM policy without credentials as in my case I just triggered the snapshot without credentials and worked fine but yet to try the restoration part.
2

Hi Team,

Can anyone help on this issue please?

3

This exception is because your OS is returning some error code (probably EPERM) when Elasticsearch tries to create this directory. It's nothing to do with SLM or user credentials, it's a system issue.

4

Hi David,

Thanks for the update.

Just wanted to know,

  1. what is the best practice to schedule the SLM policy whether scheduling them with user credentials or without user credentials.
  2. Also, whether the restoration will work fine if I schedule the SLM policy without credentials as in my case I just triggered the snapshot without credentials and worked fine but yet to try the restoration part.
5

I don't know, sorry. I guess the best practice is to do the one which works?

6

Thanks David

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.