Access_denied_exception

alexus · January 7, 2018, 3:47am

I need to look into my backups, so I'm trying to run older version of elasticsearch and register a snapshot repository and getting following error:

root@bckp:/# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "path.repo=/mnt/" -d docker.elastic.co/elasticsearch/elasticsearch:5.6.5
baa6f64799cc751ca562918f660df9d6c2e0a58f2c43e1af25272fd5c02a6376
root@bckp:/# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_backup] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_backup] failed to create repository",
    "caused_by" : {
      "type" : "access_denied_exception",
      "reason" : "/mnt/bckp"
    }
  },
  "status" : 500
}
root@bckp:/# ls -ld /mnt/ /mnt/bckp/
drwxr-xr-x  3 root root 4096 Jan  7 03:26 /mnt/
drwxr-xr-x 10 root root 4096 Jul 19 20:20 /mnt/bckp/
root@bckp:/#

Please advise.

dadoonet · January 7, 2018, 8:41am

Elasticsearch does not run as root.
Change the owner of your dirs.

HTH

alexus · January 8, 2018, 1:31am

ownership of "/mnt/bckp/backups" is identical as at the time of snapshot:

# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp/backups/"}}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[my_backup] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[my_backup] failed to create repository",
    "caused_by" : {
      "type" : "access_denied_exception",
      "reason" : "/mnt/bckp"
    }
  },
  "status" : 500
}
# ls -ld /mnt/bckp/backups/
drwxr-xr-x 4 991 991 12288 Aug  6 00:40 /mnt/bckp/backups/
# 
# mount | grep '/mnt/bckp'
/dev/mapper/vg_bckp-lv_bckp on /mnt/bckp type ext4 (rw,relatime)
#

dadoonet · January 8, 2018, 3:31am

Ok. So backups belongs to another user and this is where you are writing. That seems ok then.

What was the version you used last time you created a snapshot?

alexus · January 8, 2018, 4:34am

snapshot was created with version 5.x (possibly 5.4.x or 5.5.x; I can't recall exactly, it was created somewhere early fall of 2017 and at the time latest version of elasticsearch was used).

dadoonet · January 8, 2018, 6:03am

I was confused by

so I'm trying to run older version of elasticsearch

Is there any stacktrace in logs?

alexus · January 9, 2018, 5:18am

root@bckp:~# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -e "path.repo=/mnt/bckp/backups" --rm docker.elastic.co/elasticsearch/elasticsearch:5.6.5 
[2018-01-09T05:17:55,673][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: Unable to access 'path.repo' (/mnt/bckp/backups)
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:123) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:70) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.6.5.jar:5.6.5]
Caused by: java.lang.IllegalStateException: Unable to access 'path.repo' (/mnt/bckp/backups)
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:450) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:332) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:246) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:119) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:228) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
        ... 6 more
Caused by: java.nio.file.AccessDeniedException: /mnt/bckp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:492) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addPath(Security.java:448) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.addFilePermissions(Security.java:332) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.createPermissions(Security.java:246) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Security.configure(Security.java:119) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:228) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:342) ~[elasticsearch-5.6.5.jar:5.6.5]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:132) ~[elasticsearch-5.6.5.jar:5.6.5]
        ... 6 more
root@bckp:~#

dadoonet · January 9, 2018, 8:56am

It appears that elasticsearch process does not have write access to /mnt/bckp.

Can you change the owner of this dir?

alexus · January 9, 2018, 5:09pm

@dadoonet

I changed permissions of /mnt/bckp to 777 permission:

root@bckp:~# ls -ld /mnt/bckp/
drwxrwxrwx 11 root root 4096 Jan  8 01:25 /mnt/bckp/
root@bckp:~#

tried to restart and got exact same error...

dadoonet · January 9, 2018, 5:21pm

Exactly this?

Caused by: java.nio.file.AccessDeniedException: /mnt/bckp
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.8.0_151]
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.8.0_151]
        at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectory(Files.java:674) ~[?:1.8.0_151]
        at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781) ~[?:1.8.0_151]
        at java.nio.file.Files.createDirectories(Files.java:767) ~[?:1.8.0_151]
        at org.elasticsearch.bootstrap.Security.ensureDirectoryExists(Security.java:492) ~[elasticsearch-5.6.5.jar:5.6.5]

dadoonet · January 9, 2018, 5:22pm

If it's the case, may be something weird on a Unix level with this dir...

I have no more idea. May be @ywelsch or @Igor_Motov have an idea?

ywelsch · January 9, 2018, 5:45pm

Are you perhaps confusing /mnt/bckp/ in the container with /mnt/bckp/ outside the container?

Igor_Motov · January 9, 2018, 6:41pm

What @ywelsch said. It doesn't look like /mnt/bckp/ is actually visible inside the container and the registration fails when elastic tries to create this directory.

alexus · January 10, 2018, 10:20pm

@ywelsch uh-huh! great observation)

I added "-v /mnt/bckp/:/mnt/bckp" to docker run and now I'm able to do following:

root@bckp:~# docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -v /mnt/bckp/:/mnt/bckp/ -e "path.repo=/mnt/bckp" -d docker.elastic.co/elasticsearch/elasticsearch:5.6.5 
810a0e1db7e50583348c805fee1466c1e7bd89f372792c5d849b08b2bbef2451
root@bckp:~# curl --silent --request PUT --user elastic:changeme localhost:9200/_snapshot/my_backup?pretty --data '{"type":"fs","settings":{"location":"/mnt/bckp/backups/"}}'
{
  "acknowledged" : true
}
root@bckp:~#

Thank you!

system · February 7, 2018, 10:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.