Delete snapshot not working in Elasticsearch 1.7.3

Elastic Stack Elasticsearch
1

Hi All,

I am trying to delete old snapshots in the ES Repository by using the following curl command but I am getting below error,

Repository name:- backup
Repository path:- opt/grd/common/ES_Backup

Curl command
curl --user esadmin:XXXX -XDELETE "10.7.145.22:9200/_snapshot/backup/snapshot.10-01-2016.item"

Error in putty
[esadmin@dayrhefmtp005 ~]$ cd /opt/grd/common/ES_Backup
[esadmin@dayrhefmtp005 ES_Backup]$ ls -tlr

-rw-rw-r-- 1 esadmin dlmerds 61 Jan 9 21:40 metadata-snapshot.10-01-2016.chr
-rw-rw-r-- 1 esadmin dlmerds 192 Jan 9 21:41 snapshot-snapshot.10-01-2016.chr
-rw-rw-r-- 1 esadmin dlmerds 61 Jan 9 21:43 metadata-snapshot.10-01-2016.cval
-rw-rw-r-- 1 esadmin dlmerds 196 Jan 9 23:01 snapshot-snapshot.10-01-2016.cval
-rw-rw-r-- 1 esadmin dlmerds 61 Jan 9 23:03 metadata-snapshot.10-01-2016.item
-rw-rw-r-- 1 esadmin dlmerds 198 Jan 10 03:47 snapshot-snapshot.10-01-2016.item
-rw-rw-r-- 1 esadmin dlmerds 2420 Jan 20 03:09 metadata-snapshot_marvel
-rw-rw-r-- 1 esadmin dlmerds 200 Jan 20 03:10 snapshot-snapshot_marvel
-rw-rw-r-- 1 esadmin dlmerds 76 Jan 20 03:10 index
drwxrwxr-x 7 esadmin dlmerds 243 Jan 20 03:18 indices

[esadmin@dayrhefmtp005 ES_Backup]$ curl --user esadmin:XXXX -XDELETE "10.7.145.22:9200/_snapshot/backup/snapshot.10-01-2016.item"
{"error":"RepositoryException[[backup] failed to update snapshot in repository]; nested: FileNotFoundException[/opt/grd/common/ES_Backup/index (Permission denied)]; ","status":500}

[esadmin@dayrhefmtp005 ES_Backup]$ curl --user esadmin:XXXX -XDELETE "10.7.145.22:9200/_snapshot/backup/snapshot-snapshot.10-01-2016.item"
{"error":"SnapshotMissingException[[backup:snapshot-snapshot.10-01-2016.item] is missing]; nested: FileNotFoundException[/opt/grd/common/ES_Backup/snapshot-snapshot-snapshot.10-01-2016.item (No such file or directory)]; ","status":404}

I would like to tell I took backup in ES 1.7.2 version but now I am trying to DELETE snapshot in ES -1.7.3 and I don't think the version will creating the problem. But however I am not able to delete snapshot &

Please help me to resolve this error.

Thanks,
Ganeshbabu R

2

Is elasticsearch running under esadmin unix user?

Just to make sure, can you show what gives a GET _snapshot/backup

3

Yes @dadoonet elasticsearch was running under esadmin unix user.

Below is the response of GET_snapshot/backup

{
"backup": {
"type": "fs",
"settings": {
"compress": "true",
"location": "/opt/grd/common/ES_Backup"
}
}
}

4

I have no idea TBH. Looks strange to me. May be @Igor_Motov could have an idea?

5

Okay @dadoonet Let's see @Igor_Motov give us some input..

6

@r.ganeshbabu could you ssh to the machine where the master node is running and check how permissions in the directory /opt/grd/common/ES_Backup/ look like from that machine?

7

Hi @Igor_Motov

We have 3 physical servers and following details are nodes setup structure,

Server name -- Node Details
dayrhetfmp003 -- Master, Data
dayrhetfmp004 -- Master, Data
dayrhetfmp005 -- Master, Data, Client

For backup purpose we create folder ES_backup in the NAS (opt/grd/common) and all the machines are mounted to the NAS Path for the creation of repository to take snapshots.
Below is the permissions in the directory /opt/grd/common/ES_Backup/

-rw-rw-r-- 1 esadmin dmleogrds 61 Jan 9 21:40 metadata-snapshot.10-01-2016.chr
-rw-rw-r-- 1 esadmin dmleogrds 192 Jan 9 21:41 snapshot-snapshot.10-01-2016.chr
-rw-rw-r-- 1 esadmin dmleogrds 61 Jan 9 21:43 metadata-snapshot.10-01-2016.cval
-rw-rw-r-- 1 esadmin dmleogrds 196 Jan 9 23:01 snapshot-snapshot.10-01-2016.cval
-rw-rw-r-- 1 esadmin dmleogrds 61 Jan 9 23:03 metadata-snapshot.10-01-2016.item
-rw-rw-r-- 1 esadmin dmleogrds 198 Jan 10 03:47 snapshot-snapshot.10-01-2016.item
-rw-rw-r-- 1 esadmin dmleogrds 2420 Jan 20 03:09 metadata-snapshot_marvel
-rw-rw-r-- 1 esadmin dmleogrds 200 Jan 20 03:10 snapshot-snapshot_marvel
-rw-rw-r-- 1 esadmin dmleogrds 76 Jan 20 03:10 index
drwxrwxr-x 7 esadmin dmleogrds 243 Jan 20 03:18 indices

Thanks,
Ganeshbabu R