Repository_missing_exception with existing repo

[fetch (Ivan)]

Hi,

I'm using a simple script to make backups of my cluster. All it does:

1. Read snapshot number & increment it
2. Mount shared folder to /mnt/elkbackup/
3. curl -XPUT "http://localhost:9200/_snapshot/my_backup/snapshot_$NUM?wait_for_completion=true"
4. curl -XGET "http://localhost:9200/_snapshot/my_backup/snapshot_$NUM?pretty=true" -o "snapshot_$NUM.log"
5. umount folder

It was working fine for a while, but today I've noticed that it doesn't work anymore.

The repository exists:

curl -XGET "http://localhost:9200/_snapshot/my_backup/?pretty=true"
{
  "my_backup" : {
    "type" : "fs",
    "settings" : {
      "compress" : "true",
      "location" : "/mnt/elkbackup/my_backup"
    }
  }
}

The folder is accessible by elasticsearch user and contains some backups:

ls -la /mnt/elkbackup/my_backup
total 177
drwxr-xr-x   3 elasticsearch elasticsearch     23 Nov 28 13:58 .
drwxr-xr-x   3 root          root               3 Jul 27 18:52 ..
-rw-r--r--   1 elasticsearch elasticsearch     29 Aug 17 09:10 incompatible-snapshots
-rw-r--r--   1 elasticsearch elasticsearch  97154 Oct  5 03:34 index-6
-rw-r--r--   1 elasticsearch elasticsearch 108961 Oct 12 03:25 index-7
-rw-r--r--   1 elasticsearch elasticsearch      8 Oct 12 03:25 index.latest
drwxr-xr-x 530 elasticsearch elasticsearch    530 Oct 12 03:00 indices
-rw-r--r--   1 elasticsearch elasticsearch   5772 Sep 21 03:00 meta-4oQIJASSQTOfrFs0JKwZTw.dat
-rw-r--r--   1 elasticsearch elasticsearch   5695 Jul 27 18:54 meta-5dQLD08dQ0itR5KVVGo5fg.dat
-rw-r--r--   1 elasticsearch elasticsearch   5773 Oct  5 03:00 meta-GOzFgtkCSF-YO3aezwaeiQ.dat
-rw-r--r--   1 elasticsearch elasticsearch   5770 Sep  5 09:12 meta-PDlpsGkOS6ebbW95RccjEQ.dat
-rw-r--r--   1 elasticsearch elasticsearch   5773 Sep 28 03:00 meta-RIguMiwlRCODEH54rml_Cg.dat
-rw-r--r--   1 elasticsearch elasticsearch   5772 Oct 12 03:00 meta-VV2LWHPVRWaWAwr5_GYBXw.dat
-rw-r--r--   1 elasticsearch elasticsearch   5770 Sep 14 03:00 meta-ZqeFQRh7RNq8f8kpTLbLxw.dat
-rw-r--r--   1 elasticsearch elasticsearch   5695 Aug 17 09:21 meta-_RH4zc_zRXivCxG4yWd3og.dat
-rw-r--r--   1 elasticsearch elasticsearch   1521 Sep 21 03:27 snap-4oQIJASSQTOfrFs0JKwZTw.dat
-rw-r--r--   1 elasticsearch elasticsearch   1288 Jul 27 23:29 snap-5dQLD08dQ0itR5KVVGo5fg.dat
-rw-r--r--   1 elasticsearch elasticsearch   1575 Oct  5 03:34 snap-GOzFgtkCSF-YO3aezwaeiQ.dat
-rw-r--r--   1 elasticsearch elasticsearch   1476 Sep  5 10:34 snap-PDlpsGkOS6ebbW95RccjEQ.dat
-rw-r--r--   1 elasticsearch elasticsearch   1541 Sep 28 03:49 snap-RIguMiwlRCODEH54rml_Cg.dat
-rw-r--r--   1 elasticsearch elasticsearch   1583 Oct 12 03:25 snap-VV2LWHPVRWaWAwr5_GYBXw.dat
-rw-r--r--   1 elasticsearch elasticsearch   1504 Sep 14 03:45 snap-ZqeFQRh7RNq8f8kpTLbLxw.dat
-rw-r--r--   1 elasticsearch elasticsearch   1399 Aug 17 10:32 snap-_RH4zc_zRXivCxG4yWd3og.dat

But for some reason it can't neither create new snapshot, nor read existing:

curl -XPUT "http://localhost:9200/_snapshot/my_backup/snapshot_18?wait_for_completion=true&pretty=true"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_missing_exception",
        "reason" : "[my_backup] missing"
      }
    ],
    "type" : "repository_missing_exception",
    "reason" : "[my_backup] missing"
  },
  "status" : 404
}

curl -XGET "http://localhost:9200/_snapshot/my_backup/snapshot_12?pretty=true"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_missing_exception",
        "reason" : "[my_backup] missing"
      }
    ],
    "type" : "repository_missing_exception",
    "reason" : "[my_backup] missing"
  },
  "status" : 404
}

I know for sure that snapshot_12 exists.

Any ideas what is going on?

At this moment ELK's version is 5.6.4.

Update:

While playing around with ES, I have found that this behavior occures when the repository folder is not accessible during the ES startup (even if it's accessible at the time when snapshot creating called).

1. Is it intended? I don't see reasons why the volume with backups must be mounted all the time the server is running.

2. Success of the call curl -XGET "http://localhost:9200/_snapshot/my_backup/?pretty=true" confuses at this point. If ES failed to register repository, why doesn't it say anything when a user requests info about that repository?