Hi,
I'm using a simple script to make backups of my cluster. All it does:
- Read snapshot number & increment it
- Mount shared folder to /mnt/elkbackup/
curl -XPUT "http://localhost:9200/_snapshot/my_backup/snapshot_$NUM?wait_for_completion=true"
curl -XGET "http://localhost:9200/_snapshot/my_backup/snapshot_$NUM?pretty=true" -o "snapshot_$NUM.log"
- umount folder
It was working fine for a while, but today I've noticed that it doesn't work anymore.
The repository exists:
curl -XGET "http://localhost:9200/_snapshot/my_backup/?pretty=true"
{
"my_backup" : {
"type" : "fs",
"settings" : {
"compress" : "true",
"location" : "/mnt/elkbackup/my_backup"
}
}
}
The folder is accessible by elasticsearch user and contains some backups:
ls -la /mnt/elkbackup/my_backup
total 177
drwxr-xr-x 3 elasticsearch elasticsearch 23 Nov 28 13:58 .
drwxr-xr-x 3 root root 3 Jul 27 18:52 ..
-rw-r--r-- 1 elasticsearch elasticsearch 29 Aug 17 09:10 incompatible-snapshots
-rw-r--r-- 1 elasticsearch elasticsearch 97154 Oct 5 03:34 index-6
-rw-r--r-- 1 elasticsearch elasticsearch 108961 Oct 12 03:25 index-7
-rw-r--r-- 1 elasticsearch elasticsearch 8 Oct 12 03:25 index.latest
drwxr-xr-x 530 elasticsearch elasticsearch 530 Oct 12 03:00 indices
-rw-r--r-- 1 elasticsearch elasticsearch 5772 Sep 21 03:00 meta-4oQIJASSQTOfrFs0JKwZTw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5695 Jul 27 18:54 meta-5dQLD08dQ0itR5KVVGo5fg.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5773 Oct 5 03:00 meta-GOzFgtkCSF-YO3aezwaeiQ.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5770 Sep 5 09:12 meta-PDlpsGkOS6ebbW95RccjEQ.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5773 Sep 28 03:00 meta-RIguMiwlRCODEH54rml_Cg.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5772 Oct 12 03:00 meta-VV2LWHPVRWaWAwr5_GYBXw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5770 Sep 14 03:00 meta-ZqeFQRh7RNq8f8kpTLbLxw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 5695 Aug 17 09:21 meta-_RH4zc_zRXivCxG4yWd3og.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1521 Sep 21 03:27 snap-4oQIJASSQTOfrFs0JKwZTw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1288 Jul 27 23:29 snap-5dQLD08dQ0itR5KVVGo5fg.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1575 Oct 5 03:34 snap-GOzFgtkCSF-YO3aezwaeiQ.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1476 Sep 5 10:34 snap-PDlpsGkOS6ebbW95RccjEQ.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1541 Sep 28 03:49 snap-RIguMiwlRCODEH54rml_Cg.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1583 Oct 12 03:25 snap-VV2LWHPVRWaWAwr5_GYBXw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1504 Sep 14 03:45 snap-ZqeFQRh7RNq8f8kpTLbLxw.dat
-rw-r--r-- 1 elasticsearch elasticsearch 1399 Aug 17 10:32 snap-_RH4zc_zRXivCxG4yWd3og.dat
But for some reason it can't neither create new snapshot, nor read existing:
curl -XPUT "http://localhost:9200/_snapshot/my_backup/snapshot_18?wait_for_completion=true&pretty=true"
{
"error" : {
"root_cause" : [
{
"type" : "repository_missing_exception",
"reason" : "[my_backup] missing"
}
],
"type" : "repository_missing_exception",
"reason" : "[my_backup] missing"
},
"status" : 404
}
curl -XGET "http://localhost:9200/_snapshot/my_backup/snapshot_12?pretty=true"
{
"error" : {
"root_cause" : [
{
"type" : "repository_missing_exception",
"reason" : "[my_backup] missing"
}
],
"type" : "repository_missing_exception",
"reason" : "[my_backup] missing"
},
"status" : 404
}
I know for sure that snapshot_12 exists.
Any ideas what is going on?
At this moment ELK's version is 5.6.4.
Update:
While playing around with ES, I have found that this behavior occures when the repository folder is not accessible during the ES startup (even if it's accessible at the time when snapshot creating called).
-
Is it intended? I don't see reasons why the volume with backups must be mounted all the time the server is running.
-
Success of the call
curl -XGET "http://localhost:9200/_snapshot/my_backup/?pretty=true"
confuses at this point. If ES failed to register repository, why doesn't it say anything when a user requests info about that repository?