Snapshots are failing

[SteveHodges]

After a network outage, the elasticsearch snapshot is failing. After I restored the connection to the NAS I am getting errors when I try to take a snapshot.

The error is access_denied_exception

Here is the command I run to verify the snapshots and the error I get

curl -X POST "192.168.1.8:9200/_snapshot/DM_backup/_verify"

{"error":{"root_cause":[{"type":"exception","reason":"failed to create blob container"}],"type":"exception","reason":"failed to create blob container","caused_by":{"type":"access_denied_exception","reason":"/mnt/nasbackup/dm-logserver/NAS_backup_location/tests-Ly03ASkJR8WkWUrA4UAflg"}},"status":500}

I also tried to delete the oldest snapshot.

sudo curl -X DELETE "10.134.90.8:9200/_snapshot/DM_backup/snapshot_5?pretty"
Output from trying to delete the a snapshot

{
  "error" : {
    "root_cause" : [
      {
        "type" : "access_denied_exception",
        "reason" : "/mnt/nasbackup/dm-logserver/NAS_backup_location/pending-index-47-TiO4j11eTYChkZ0iw8MLHw"
      }
    ],
    "type" : "access_denied_exception",
    "reason" : "/mnt/nasbackup/dm-logserver/NAS_backup_location/pending-index-47-TiO4j11eTYChkZ0iw8MLHw"
  },
  "status" : 500
}

The snapshot is a NAS which is mounted

mount.cifs -v //192.168.2.203/public /mnt/nasbackup -o vers=1.0,rw,username=user1,password=*******

I have tried to change ownership to 777 as test. although the command is successful, the permissions don't change.

total 1031
drwxr-xr-x 2 root root      0 Jan 31  2021 .
drwxr-xr-x 2 root root      0 Jan 31  2021 ..
-rwxr-xr-x 1 root root 137326 Dec 17 15:26 index-32
-rwxr-xr-x 1 root root      8 Dec 17 15:26 index.latest
drwxr-xr-x 2 root root      0 Dec 17 15:08 indices
-rwxr-xr-x 1 root root 171992 Dec 17 15:26 meta-6n4XFTmnRkiuto2guP6lKA.dat
-rwxr-xr-x 1 root root 171992 Dec  3 08:49 meta-KnEHI40CReeWfmFPxla1QQ.dat
-rwxr-xr-x 1 root root 171992 Nov 30 12:39 meta-U0KwAyv2QkGFXIQtxKsARQ.dat
-rwxr-xr-x 1 root root 171992 Dec 10 13:02 meta-vjh2h4I-RBW5WN93n1zBcw.dat
-rwxr-xr-x 1 root root 171992 Nov 13 10:33 meta-wvy7ALr-Qq6Kiqo5-4A5OA.dat
-rwxr-xr-x 1 root root  10887 Dec 17 15:26 snap-6n4XFTmnRkiuto2guP6lKA.dat
-rwxr-xr-x 1 root root  11056 Dec  3 08:49 snap-KnEHI40CReeWfmFPxla1QQ.dat
-rwxr-xr-x 1 root root  11659 Nov 30 12:39 snap-U0KwAyv2QkGFXIQtxKsARQ.dat
-rwxr-xr-x 1 root root  11639 Dec 10 13:02 snap-vjh2h4I-RBW5WN93n1zBcw.dat
-rwxr-xr-x 1 root root  11032 Nov 13 10:33 snap-wvy7ALr-Qq6Kiqo5-4A5OA.dat

Any advice would be appreciated.

[warkolm (Mark Walkom)]

Have you tried remounting the FS?

[SteveHodges]

Yes, I have tried that already. FYI, I can create a text file as root.

[DavidTurner]

The files in the repository appear to be owned by root and apparently cannot be modified by other users. You need to adjust something so that they become writable by the user as which Elasticsearch is running (which is not root). Unfortunately I've no idea what that "something" is, it will depend on how your NAS is configured and how it's mounted.

[SteveHodges]

Thank you. This is what I thought, but I wanted to make sure.

How I am mounting is show in the post if that helps.