Snapshot repo.path deson't match. It recongized by settings api

[naoko]

I tried to create snapshot rep but failing with

location [/data/elasticsearch_backups] doesn't match any of the locations specified by path.repo

config is there:

cat /etc/elasticsearch/elasticsearch.yml | grep repo
        path.repo: /data/elasticsearch_backups

Also, this config is recongized via node settings api:

    curl -s http://192.168.51.80:9200/_nodes/settings?pretty | grep 'repo\|host"'
          "host" : "192.168.51.80",
              "host" : "192.168.51.80"
              "repo" : "/data/elasticsearch_backups"

and the mounted share directory have read/write for elasticsearch

    ls -al /data/ | grep elastic
    drwxr-xr-x  8 elasticsearch elasticsearch  114 May 14  2019 elasticsearch_backups

but this is what I get... and I'm out of idea... any advice would be appriciated!

    http://192.168.51.80:9200/_snapshot/_all?pretty

    curl -XPUT -H "Content-Type: application/json" 'http://192.168.51.81:9200/_snapshot/es6_dev' -d '{
      "type": "fs",
      "settings": {
         "location": "/data/elasticsearch_backups",
         "compress": true
      }
    }'

    {"error":{"root_cause":[{"type":"repository_exception","reason":"[es6_dev] location [/data/elasticsearch_backups] doesn't match any of the locations specified by path.repo"}],"type":"repository_exception","reason":"[es6_dev] failed to create repository","caused_by":{"type":"repository_exception","reason":"[es6_dev] location [/data/elasticsearch_backups] doesn't match any of the locations specified by path.repo"}},"status":500}

[ropc (Romain Chanu)]

Hi @naoko

1. How many nodes are running in the cluster? Which version of ES are you running?

2. Could you set path.repo in all your ES nodes as followed:

path.repo: ["/data/elasticsearch_backups"]

and restart all the nodes. Does that solve the problem?

Thank you.

[naoko]

I am not allowed to restart nodes on this cluster but there are 10 nodes on the cluster. on my local, path.repo value being string worked (instead of list of string) so I'm not sure that's the issue. ES version is 6.8.
One thing though... although all 10 clusters share single mounted volume, the mounted name is different. Would that cause the issue?

[ropc (Romain Chanu)]

@naoko From our documentation: " The shared file system repository ( "type": "fs" ) uses the shared file system to store snapshots. In order to register the shared file system repository it is necessary to mount the same shared filesystem to the same location on all master and data nodes. This location (or one of its parent directories) must be registered in the path.repo setting on all master and data nodes.

If could set the same value in all the required nodes and restart these nodes (any changes in elasticsearch.yml require a restart), then let's see if that solves the problem.

Thank you.