After creating the snapshot, getting snapshot_missing_exception and no_such_file_exception

[Ramesh_Perumal]

Hi All,

I have created the repository to create snapshot with the following steps in two node (machine) cluster:

1. curl -XPUT "https://:9200$HOSTNAME/_snapshot/test13?verify=false" -H 'Content-Type: application/json' -d' { "type": "fs", "settings": { "location": "/msv_backup/ESBackup/counters/", "compress": true } } '
   {"acknowledged":true}

2. curl -X PUT "https://:9200$HOSTNAME/_snapshot/test13/my_test13?wait_for_completion=true&pretty" -H 'Content-Type: application/json' -d'{ "indices": "counters-2023.04.14-60m,counters-2023.04.14-5m", "ignore_unavailable": true, "include_global_state": false}'
   {
   "snapshot" : {
   "snapshot" : "my_test13",
   "uuid" : "x_edhKvJQRyb9TE1nejxrg",
   "version_id" : 135248427,
   "version" : "1.3.6",
   "indices" : [
   "counters-2023.04.14-60m",
   "counters-2023.04.14-5m"
   ],
   "data_streams" : ,
   "include_global_state" : false,
   "state" : "SUCCESS",
   "start_time" : "2023-04-14T05:50:58.265Z",
   "start_time_in_millis" : 1681451458265,
   "end_time" : "2023-04-14T05:50:58.265Z",
   "end_time_in_millis" : 1681451458265,
   "duration_in_millis" : 0,
   "failures" : ,
   "shards" : {
   "total" : 2,
   "failed" : 0,
   "successful" : 2
   }
   }
   }

3. curl -XGET "https://:9200$HOSTNAME/_snapshot/test13/my_test13?pretty"
   {
   "snapshots" : [
   {
   "snapshot" : "my_test13",
   "uuid" : "x_edhKvJQRyb9TE1nejxrg",
   "version_id" : 135248427,
   "version" : "1.3.6",
   "indices" : [
   "counters-2023.04.14-60m",
   "counters-2023.04.14-5m"
   ],
   "data_streams" : ,
   "include_global_state" : false,
   "state" : "SUCCESS",
   "start_time" : "2023-04-14T05:50:58.265Z",
   "start_time_in_millis" : 1681451458265,
   "end_time" : "2023-04-14T05:50:58.265Z",
   "end_time_in_millis" : 1681451458265,
   "duration_in_millis" : 0,
   "failures" : ,
   "shards" : {
   "total" : 2,
   "failed" : 0,
   "successful" : 2
   }
   }
   ]
   }

4. curl -XGET "https://:9200$HOSTNAME/_snapshot/test13/my_test13/_status?pretty"
   {
   "error" : {
   "root_cause" : [
   {
   "type" : "snapshot_missing_exception",
   "reason" : "[test13:my_test13/x_edhKvJQRyb9TE1nejxrg] is missing"
   }
   ],
   "type" : "snapshot_missing_exception",
   "reason" : "[test13:my_test13/x_edhKvJQRyb9TE1nejxrg] is missing",
   "caused_by" : {
   "type" : "no_such_file_exception",
   "reason" : "/backup/ESBackup/counters/indices/Vrxkp7oiSQ6xwQFMiWTxog/0/snap-x_edhKvJQRyb9TE1nejxrg.dat"
   }
   },
   "status" : 404
   }
   Due to the above issue, not able to restore the snapshot, since I'm getting file is missing. But file is there in the machine:
   ll /msv_backup/ESBackup/counters/indices/Vrxkp7oiSQ6xwQFMiWTxog/0/snap-x_edhKvJQRyb9TE1nejxrg.dat
   -rw-r--r--. 1 elasticsearch elasticsearch 1286 Apr 14 07:50 /msv_backup/ESBackup/counters/indices/Vrxkp7oiSQ6xwQFMiWTxog/0/snap-x_edhKvJQRyb9TE1nejxrg.dat

Please share your expertise.

Regards,
Ramesh P

[Christian_Dahlqvist]

@Ramesh_Perumal

curl -XPUT "https://:9200$HOSTNAME/_snapshot/test13?verify=false"

Why are you setting verify to false? Is this a shared filesystem, e.g. an NFS mount?