Hi team,
I'm trying to create a snapshot from the cluster and i am getting some errors.
ELK Stack: 7.14.1
Elasticsearch Nodes: 4
First i created the snapshot repository:
PUT /_snapshot/backup_1
{
"type": "fs",
"settings": {
"location": "/etc/elasticsearch/snapshots"
}
}
Second, i create the snapshot :
PUT /_snapshot/backup_1/snapshot_1?wait_for_completion=true&pretty
But I am getting this error:
{
"error" : {
"root_cause" : [
{
"type" : "snapshot_exception",
"reason" : "[backup_1:snapshot_1/0531DoxhR82OP7f3TFm5bA] failed to update snapshot in repository"
}
],
"type" : "snapshot_exception",
"reason" : "[backup_1:snapshot_1/0531DoxhR82OP7f3TFm5bA] failed to update snapshot in repository",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/meta-0531DoxhR82OP7f3TFm5bA.dat",
"suppressed" : [
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
},
{
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/snap-0531DoxhR82OP7f3TFm5bA.dat"
},
{
"type" : "exception",
"reason" : "failed to create blob container",
"caused_by" : {
"type" : "access_denied_exception",
"reason" : "/etc/elasticsearch/snapshots/indices"
}
}
]
}
},
"status" : 500
}
This is the info of snapshot repositories that i have created:
{
"my_backup" : {
"type" : "fs",
"settings" : {
"location" : "my_backup_location"
}
},
"snapshots" : {
"type" : "fs",
"settings" : {
"location" : "/etc/elasticsearch/snapshots"
}
},
"backup_1" : {
"type" : "fs",
"settings" : {
"location" : "/etc/elasticsearch/snapshots/"
}
}
}
This are the permissions of the elasticsearch folder:
drwxr-s--- 4 root elasticsearch 4096 Sep 19 10:13 elasticsearch/
And this are the permissions folder in every elasticsearch node:
drwxrwsrwx 3 root elasticsearch 4096 Sep 21 09:33 snapshots/
Inside /etc/elasticsearch/snapshots there is a second folder called indices
drwxrwsrwx 14 elasticsearch elasticsearch 4096 Sep 21 09:59 indices/
Just mention that /etc/elasticsearch/snapshots is a folder created manually in every node. Also, as you can see, to avoid issues with permissions i setup the folders to 777.
These are the logs from /var/log/elasticsearch/cluster.log
at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
... 8 more
Suppressed: org.elasticsearch.ElasticsearchException: failed to create blob container
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:56) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1599) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1450) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/indices
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:396) ~[?:?]
at java.nio.file.Files.createDirectory(Files.java:694) ~[?:?]
at java.nio.file.Files.createAndCheckIsDirectory(Files.java:801) ~[?:?]
at java.nio.file.Files.createDirectories(Files.java:787) ~[?:?]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.buildAndCreate(FsBlobStore.java:68) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:54) ~[elasticsearch-7.14.1.jar:7.14.1]
... 8 more
Suppressed: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/snap-0531DoxhR82OP7f3TFm5bA.dat
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219) ~[?:?]
at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478) ~[?:?]
at java.nio.file.Files.newOutputStream(Files.java:224) ~[?:?]
at org.elasticsearch.common.blobstore.fs.FsBlobContainer$BlobOutputStream.<init>(FsBlobContainer.java:358) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeToPath(FsBlobContainer.java:276) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:267) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.write(ChecksumBlobStoreFormat.java:275) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$47(BlobStoreRepository.java:1467) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
Suppressed: org.elasticsearch.ElasticsearchException: failed to create blob container
at org.elasticsearch.common.blobstore.fs.FsBlobStore.blobContainer(FsBlobStore.java:56) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.indexContainer(BlobStoreRepository.java:1599) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.lambda$finalizeSnapshot$46(BlobStoreRepository.java:1450) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:34) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:732) ~[elasticsearch-7.14.1.jar:7.14.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-7.14.1.jar:7.14.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/snapshots/indices
at sun.nio.fs.UnixException.translateToIOException(UnixExcep
Ans this is the path specified in elasticsearch.yml file. All the nodes include the specified path.repo
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
path.repo: /etc/elasticsearch/snapshots
And this is the status of the cluster:
{
"cluster_name" : "lx-z-elk",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 4,
"number_of_data_nodes" : 4,
"active_primary_shards" : 12,
"active_shards" : 24,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
Any idea about this issue?
Thank you in advance.