Can't create snapshots after restoring data

Elastic Stack Elasticsearch
1

The elasticseach version is 6.2.2. Today I encountered an error when trying to create a snapshot after restoring data. Can anyone please provide any advices?

The test steps are as below,

  1. Start an elasticsearch instance, and attach a volume to /usr/share/elasticsearch/data;
  2. Generate some log data;
  3. Create a repository using the following command successfully;

curl -X PUT -H "Content-Type:application/json" -d '{"type":"fs", "settings":{"location": "/usr/share/elasticsearch/data/backup"}}' http://10.97.116.168:9200/_snapshot/ben_backup

  1. Create a snapshot using the following command successfully;

curl -X PUT http://10.97.116.168:9200/_snapshot/ben_backup/snap_1

  1. Shutdown the elasticsearch instance;

  2. Start a new elasticsearch instance on other node, and attached the same volume to /usr/share/elasticsearch/data;

  3. Create a repository using the following command successfully;

curl -X PUT -H "Content-Type:application/json" -d '{"type":"fs", "settings":{"location": "/usr/share/elasticsearch/data/backup"}}' http://10.101.44.198:9200/_snapshot/ben_backup

  1. Restore the snapshot using the following command successfully;

curl -X POST http://10.101.44.198:9200/_snapshot/ben_backup/snap_1/_restore

  1. Feed more log data into elasticsearch successfully;

  2. Try to generate a new snapshot using the following command, but failed this time.
    curl -X PUT http://10.101.44.198:9200/_snapshot/ben_backup/snap_2

The error response what I got was as below,

{
  "error": {
    "root_cause": [
      {
        "type": "snapshot_creation_exception",
        "reason": "[ben_backup:snap_2/eBS31A1mSFG1Ffu8FQmgRw] failed to create snapshot"
      }
    ],
    "type": "snapshot_creation_exception",
    "reason": "[ben_backup:snap_2/eBS31A1mSFG1Ffu8FQmgRw] failed to create snapshot",
    "caused_by": {
      "type": "access_denied_exception",
      "reason": "/usr/share/elasticsearch/data/backup/indices/Rd6MzRfdTiOOg6N0UIEdwg/meta-eBS31A1mSFG1Ffu8FQmgRw.dat"
    }
  },
  "status": 500
}
2

Elasticsearch generated the following exceptions,

2018-07-31T09:39:25,272][WARN ][o.e.s.SnapshotsService   ] [t-hprrzcjx5s27dyucxyux-es-b9f74b99f-h6xwp-default] failed to create snapshot [snap_2/eBS31A1mSFG1Ffu8FQmgRw]
org.elasticsearch.snapshots.SnapshotCreationException: [ben_backup:snap_2/eBS31A1mSFG1Ffu8FQmgRw] failed to create snapshot
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:335) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:364) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$700(SnapshotsService.java:105) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService$1.lambda$clusterStateProcessed$1(SnapshotsService.java:282) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/backup/indices/Rd6MzRfdTiOOg6N0UIEdwg/meta-eBS31A1mSFG1Ffu8FQmgRw.dat
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) ~[?:1.8.0_161]
	at java.nio.file.Files.newOutputStream(Files.java:216) ~[?:1.8.0_161]
	at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:130) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.writeBlob(ChecksumBlobStoreFormat.java:191) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.write(ChecksumBlobStoreFormat.java:161) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:332) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 7 more
[2018-07-31T09:39:25,280][WARN ][r.suppressed             ] path: /_snapshot/ben_backup/snap_2, params: {repository=ben_backup, snapshot=snap_2}
org.elasticsearch.snapshots.SnapshotCreationException: [ben_backup:snap_2/eBS31A1mSFG1Ffu8FQmgRw] failed to create snapshot
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:335) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService.beginSnapshot(SnapshotsService.java:364) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService.access$700(SnapshotsService.java:105) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.snapshots.SnapshotsService$1.lambda$clusterStateProcessed$1(SnapshotsService.java:282) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: java.nio.file.AccessDeniedException: /usr/share/elasticsearch/data/backup/indices/Rd6MzRfdTiOOg6N0UIEdwg/meta-eBS31A1mSFG1Ffu8FQmgRw.dat
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:?]
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) ~[?:?]
	at java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:434) ~[?:1.8.0_161]
	at java.nio.file.Files.newOutputStream(Files.java:216) ~[?:1.8.0_161]
	at org.elasticsearch.common.blobstore.fs.FsBlobContainer.writeBlob(FsBlobContainer.java:130) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.writeBlob(ChecksumBlobStoreFormat.java:191) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.ChecksumBlobStoreFormat.write(ChecksumBlobStoreFormat.java:161) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.initializeSnapshot(BlobStoreRepository.java:332) ~[elasticsearch-6.2.2.jar:6.2.2]
	... 7 more
3

I changed the permission of the fs from 755 to 777, and it worked. I don't understand why I could create the snapshot successfully in the first time, but failed in the second elasticsearch instance.

4

path.repo: [/data/sdd1/esbackup]
backup directory need share fs as nfs ,not local fs.

5

What I used was a shared volume.

6

each node have access wirte and read data with elasticsearch user ?
can write and read ?
test by create file and write file change file or delete file.

7

Previously the mod of the shared volume was 755, and it could be read/written by the first elasticsearch instance with elasticsearch user.

But it could only be ready by the second elasticsearch instance with user elasticsearch, and couldn't be written. After changing the mod to 777, then it worked. So it's a little confusing why it could be written by the first one, but couldn't by the second.

8

each node show the elasitcsearch user uid.
maybe not same.

9

@zqc0512 Thanks for the response.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.