Access_denied_exception

alexus · January 7, 2018, 3:47am

I need to look into my backups, so I'm trying to run older version of elasticsearch and register a snapshot repository and getting following error:

root@bckp:/# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "path.repo=/mnt/" -d docker.elastic.co/elasticsearch/elasticsearch:5.6.5
baa6f64799cc751ca562918f660df9d6c2e0a58f2c43e1af25272fd5c02a6376
root@bckp:/# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_backup] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_backup] failed to create repository",
    "caused_by" : {
      "type" : "access_denied_exception",
      "reason" : "/mnt/bckp"
    }
  },
  "status" : 500
}
root@bckp:/# ls -ld /mnt/ /mnt/bckp/
drwxr-xr-x  3 root root 4096 Jan  7 03:26 /mnt/
drwxr-xr-x 10 root root 4096 Jul 19 20:20 /mnt/bckp/
root@bckp:/#

Please advise.

dadoonet · January 7, 2018, 8:41am

Elasticsearch does not run as root.
Change the owner of your dirs.

HTH

alexus · January 8, 2018, 1:31am

ownership of "/mnt/bckp/backups" is identical as at the time of snapshot:

# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp/backups/"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_backup] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_backup] failed to create repository",
    "caused_by" : {
      "type" : "access_denied_exception",
      "reason" : "/mnt/bckp"
    }
  },
  "status" : 500
}
# ls -ld /mnt/bckp/backups/
drwxr-xr-x 4 991 991 12288 Aug  6 00:40 /mnt/bckp/backups/
# 
# mount | grep '/mnt/bckp'
/dev/mapper/vg_bckp-lv_bckp on /mnt/bckp type ext4 (rw,relatime)
#

dadoonet · January 8, 2018, 3:31am

Ok. So backups belongs to another user and this is where you are writing. That seems ok then.

What was the version you used last time you created a snapshot?

alexus · January 8, 2018, 4:34am

snapshot was created with version 5.x (possibly 5.4.x or 5.5.x; I can't recall exactly, it was created somewhere early fall of 2017 and at the time latest version of elasticsearch was used).

dadoonet · January 8, 2018, 6:03am

I was confused by

so I'm trying to run older version of elasticsearch

Is there any stacktrace in logs?

alexus · January 9, 2018, 5:18am

root@bckp:~# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "path.repo=/mnt/bckp/backups" --rm docker.elastic.co/elasticsearch/elasticsearch:5.6.5 
[2018-01-09T05:17:55,673][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/mnt/bckp/backups)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.5.jar:5.6.5]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/mnt/bckp/backups)
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:450) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:332) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:246) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:119) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:228) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /mnt/bckp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:492) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:448) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:332) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:246) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:119) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:228) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
        ... 6 more
root@bckp:~#

dadoonet · January 9, 2018, 8:56am

It appears that elasticsearch process does not have write access to /mnt/bckp.

Can you change the owner of this dir?

alexus · January 9, 2018, 5:09pm

@dadoonet

I changed permissions of /mnt/bckp to 777 permission:

root@bckp:~# ls -ld /mnt/bckp/
drwxrwxrwx 11 root root 4096 Jan  8 01:25 /mnt/bckp/
root@bckp:~#

tried to restart and got exact same error...

dadoonet · January 9, 2018, 5:21pm

Exactly this?

Caused by: java.nio.file.AccessDeniedException: /mnt/bckp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:492) ~[elasticsearch-5.6.5.jar:5.6.5]

dadoonet · January 9, 2018, 5:22pm

If it's the case, may be something weird on a Unix level with this dir...

I have no more idea. May be @ywelsch or @Igor_Motov have an idea?

ywelsch · January 9, 2018, 5:45pm

Are you perhaps confusing /mnt/bckp/ in the container with /mnt/bckp/ outside the container?

Igor_Motov · January 9, 2018, 6:41pm

What @ywelsch said. It doesn't look like /mnt/bckp/ is actually visible inside the container and the registration fails when elastic tries to create this directory.

alexus · January 10, 2018, 10:20pm

@ywelsch uh-huh! great observation)

I added "-v /mnt/bckp/:/mnt/bckp" to docker run and now I'm able to do following:

root@bckp:~# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -v /mnt/bckp/:/mnt/bckp/ -e "path.repo=/mnt/bckp" -d docker.elastic.co/elasticsearch/elasticsearch:5.6.5 
810a0e1db7e50583348c805fee1466c1e7bd89f372792c5d849b08b2bbef2451
root@bckp:~# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp/backups/"}}'
{
  "acknowledged" : true
}
root@bckp:~#

Thank you!

system · February 7, 2018, 10:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to create the repository for snapshots after mounting nfs share in the container Elasticsearch docker , snapshot-and-restore	7	454	September 14, 2022
Create Repository access denied Elasticsearch	3	1548	July 5, 2017
Having trouble creating Elasticsearch Snapshot Repository Elasticsearch docker , snapshot-and-restore	10	644	September 15, 2022
Elasticsearch with non-root user - Backup restore is not working Elasticsearch docker , snapshot-and-restore	3	532	December 22, 2021
Elasticsearch running on Docker failing in taking snapshots and restore Elasticsearch docker	7	1071	April 28, 2020

Access_denied_exception

Related topics