Failed to create snapshot

Hi team,

I'm trying to create a snapshot from the cluster and i am getting some errors.

ELK Stack: 7.14.1
Elasticsearch Nodes: 4

First i created the snapshot repository:

PUT /_snapshot/backup_1
{
  "type": "fs",
  "settings": {
    "location": "/etc/elasticsearch/snapshots"
  }
}

Second, i create the snapshot :

PUT /_snapshot/backup_1/snapshot_1?wait_for_completion=true&pretty

But I am getting this error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "snapshot_exception",
        "reason" : "[backup_1:snapshot_1/0531DoxhR82OP7f3TFm5bA] failed to update snapshot in repository"
      }
    ],
    "type" : "snapshot_exception",
    "reason" : "[backup_1:snapshot_1/0531DoxhR82OP7f3TFm5bA] failed to update snapshot in repository",
    "caused_by" : {
      "type" : "access_denied_exception",
      "reason" : "/etc/elasticsearch/snapshots/meta-0531DoxhR82OP7f3TFm5bA.dat",
      "suppressed" : [
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        },
        {
          "type" : "access_denied_exception",
          "reason" : "/etc/elasticsearch/snapshots/snap-0531DoxhR82OP7f3TFm5bA.dat"
        },
        {
          "type" : "exception",
          "reason" : "failed to create blob container",
          "caused_by" : {
            "type" : "access_denied_exception",
            "reason" : "/etc/elasticsearch/snapshots/indices"
          }
        }
      ]
    }
  },
  "status" : 500
}

This is the info of snapshot repositories that i have created:

{
  "my_backup" : {
    "type" : "fs",
    "settings" : {
      "location" : "my_backup_location"
    }
  },
  "snapshots" : {
    "type" : "fs",
    "settings" : {
      "location" : "/etc/elasticsearch/snapshots"
    }
  },
  "backup_1" : {
    "type" : "fs",
    "settings" : {
      "location" : "/etc/elasticsearch/snapshots/"
    }
  }
}

This are the permissions of the elasticsearch folder:

drwxr-s---  4 root elasticsearch  4096 Sep 19 10:13 elasticsearch/

And this are the permissions folder in every elasticsearch node:

drwxrwsrwx  3 root elasticsearch  4096 Sep 21 09:33 snapshots/

Inside /etc/elasticsearch/snapshots there is a second folder called indices

drwxrwsrwx 14 elasticsearch elasticsearch 4096 Sep 21 09:59 indices/

Just mention that /etc/elasticsearch/snapshots is a folder created manually in every node. Also, as you can see, to avoid issues with permissions i setup the folders to 777.

These are the logs from /var/log/elasticsearch/cluster.log

                at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
                at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
                at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
                ... 8 more
        Suppressed: org.elasticsearch.ElasticsearchException: failed to create blob container
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:56) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1599) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1450) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
                at java.lang.Thread.run(Thread.java:831) [?:?]
        Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/indices
                at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
                at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:396) ~[?:?]
                at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
                at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
                at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
                ... 8 more
        Suppressed: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/snap-0531DoxhR82OP7f3TFm5bA.dat
                at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
                at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
                at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
                at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478) ~[?:?]
                at java.nio.file.Files.newOutputStream(Files.java:224) ~[?:?]
                at org.elasticsearch.common.blobstore.fs.FsBlobContainer$BlobOutputStream.<init>(FsBlobContainer.java:358) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeToPath(FsBlobContainer.java:276) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:267) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.write(ChecksumBlobStoreFormat.java:275) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$47(BlobStoreRepository.java:1467) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
                at java.lang.Thread.run(Thread.java:831) [?:?]
        Suppressed: org.elasticsearch.ElasticsearchException: failed to create blob container
                at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:56) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1599) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1450) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
                at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
                at java.lang.Thread.run(Thread.java:831) [?:?]
        Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/indices
                at sun.nio.fs.UnixException.translateToIOException(UnixExcep

Ans this is the path specified in elasticsearch.yml file. All the nodes include the specified path.repo

# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
path.repo: /etc/elasticsearch/snapshots

And this is the status of the cluster:

{
  "cluster_name" : "lx-z-elk",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 4,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 12,
  "active_shards" : 24,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Any idea about this issue?

Thank you in advance.

Snapshots require a shared filesystem accessible to all master and data nodes in the same path. Having local directories on the machines with the same path will therefore not work.

Thanks for your answer @Christian_Dahlqvist

So is there any way to create something locally in order to test the snapshot functionality?

If not, is there any tutorial to create a blob storage in Azure ?

Thanks in advance.

Not unless you have a single node.

Check out Azure Repository Plugin | Elasticsearch Plugins and Integrations [7.14] | Elastic

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.