Snapshot Repositories are all unwriteable when created in Elasticsearch, they do not inherit parent permissions and have a weird owner

Elastic Stack Elasticsearch
1

Hello!

Here's a riddle:
I have two Dockered Elastic stacks on two different networks: Stack A and Stack B.
Both have a CIFS backup mounted with the same /etc/fstab line. Both use the same docker-compose lines to mount the shares as volumes. Both CIFS shares are hosted on the same NAS hardware (as in, two identical NAS devices with the same permissions). Also, the elasticsearch.yml is the same settings wise.

When I create a backup Repository in Kibana or using the console:

  • Stack A creates a directory with 0777 permissions owned by root on the CIFS share and can then do snapshots to it.
  • Stack B creates a directory with 0775 permissions owned by 1030 on the CIFS share and cannot do anything with it.

Error log on Stack B:

{
  "error": {
    "root_cause": [
      {
        "type": "exception",
        "reason": "failed to create blob container"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[test] path  is not accessible on master node",
    "caused_by": {
      "type": "exception",
      "reason": "failed to create blob container",
      "caused_by": {
        "type": "access_denied_exception",
        "reason": "/usr/share/elasticsearch/snapshots/test/tests-Vm9gr8K5RwuCTlnoNjHrlw"
      }
    }
  },
  "status": 500
}

Stack A repository directory ls

user@hostname1:/# ls /mnt/ESBKUP/ -l
total 0
drwxrwxrwx 3 root users 0 Sep  8 16:30 7.6.0
drwxrwxrwx 3 root users 0 Sep  8 16:35 7.6.1
drwxrwxrwx 3 root users 0 Sep  4 11:30 7.8.1
drwxrwxrwx 3 root users 0 Oct  7 09:42 7.9.0
drwxrwxrwx 6 root users 0 Mar 18  2020 config

Stack B repository directory ls

user@hostname2:/# ls /mnt/ESBKUP/ -l
total 1
drwxrwxr-x 2 1030 users  0 Oct  7 10:51 fffff
-rw-r--r-- 1 1030 users 32 Sep  8 16:13 hello.txt
drwxrwxr-x 2 1030 users  0 Oct  7  2020 test
drwxrwxr-x 2 1030 users  0 Oct  7 11:05 testrepo

(fffff, test and testrepo are the repositories created in Kibana/Elastic)

Can anybody think why Stack B isn't inheriting the directory permissions (0777) on the parent docker volume when creating a repo? And who is user 1030?? This uid doesnt exist on the host or container..

In reality I think 0775 doesnt work as docker volumes are mounted as root and this cannot be changed in assured production. But I never had an issue on Stack A as it always created the repo's with 0777.

Many thanks,

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.