hi
I am using elasticsearch 6.8.22 with plugin repository-s3
I also use ceph s3 storage running on rook(k8s)
I'm trying to set up a snapshot backup for it:
I create a repository command:
curl -sk -X PUT "https://localhost:9200/_snapshot/ceph?pretty" -H 'Content-Type: application/json' -d'
{
"type": "s3",
"settings": {
"bucket": "bucket_name",
"endpoint": "ceph-s3.hostname.com:443",
"protocol": "https",
"region": "region_name",
"base_path": "bucket_name",
"read_timeout": "3600s",
"disable_chunked_encoding": true
}
}'
answer:
{
"acknowledged" : true
}
Everything went well and I can start backup:
now=`date +%Y%m%d`
curl -k -X PUT 'https://localhost:9200/_snapshot/ceph/'"$now"'?wait_for_completion=true&pretty' -H 'Content-Type: application/json' -d' {
"indices": "bases_2"
}'
answer:
{
"snapshot" : {
"snapshot" : "20211223",
"uuid" : "8SMS5XFKQtiSW1jKsjEyzA",
"version_id" : 6082299,
"version" : "6.8.22",
"indices" : [
"bases_2"
],
"include_global_state" : true,
"state" : "SUCCESS",
"start_time" : "2021-12-23T16:14:03.709Z",
"start_time_in_millis" : 1640276043709,
"end_time" : "2021-12-23T16:14:18.560Z",
"end_time_in_millis" : 1640276058560,
"duration_in_millis" : 14851,
"failures" : [ ],
"shards" : {
"total" : 5,
"failed" : 0,
"successful" : 5
}
}
}
In the master node logs:
[2021-12-23T19:10:13,759][INFO ][o.e.s.SnapshotsService ] [node-01] snapshot [ceph:20211223/dCAQNUiHQDeTPXjl3iUGHw] started
[2021-12-23T19:10:32,625][INFO ][o.e.s.SnapshotsService ] [node-01] snapshot [ceph:20211223/dCAQNUiHQDeTPXjl3iUGHw] completed with state [SUCCESS]
But I cannot see the created snapshot in the snapshot listing
curl -X GET 'https://localhost:9200/_cat/snapshots/ceph?v'
id status start_epoch start_time end_epoch end_time duration indices successful_shards ailed_shards total_shards
..empty output..
The state of s3 repository immediately after the completion of the snapshot:
$ aws --profile=ceph --endpoint=https://ceph-s3.hostname.com s3 ls bucket_name/
PRE indices/
PRE tests-Vf_GWcT0TsWuTNoLuf2mxg/
PRE tests-gIQqkXEiSo2AjQr87IJIPw/
PRE tests-ijVHsV_mTa-UNKUq2_k5wA/
2021-12-23 19:14:19 177 index-0
2021-12-23 19:14:20 8 index.latest
2021-12-23 19:14:03 27495 meta-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:12 27495 meta-dCAQNUiHQDeTPXjl3iUGHw.dat
2021-12-23 19:14:18 249 snap-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:30 249 snap-dCAQNUiHQDeTPXjl3iUGHw.dat
In the logs of s3 ceph (ingress) access when requesting a snapshot listing, I see a file access:
{
"protocol": "HTTP/1.1",
"path": "/bucket_name/index--1",
"bytes_sent": 213,
"bytes_received": 0,
"user_agent": "aws-sdk-java/1.11.406 Linux/5.4.0-65-generic OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282",
"method": "GET",
"upstream_cluster": "outbound|80||rook-ceph-rgw-gib.rook-ceph.svc.cluster.local",
"response_code_details": "via_upstream",
"upstream_transport_failure_reason": null,
"response_code": 404,
"referer": null
}
(which is not on my server) (!)
I can see the index-0 file in the s3 ceph repository.
I copy it to the index - 1 file:
aws --profile=ceph --endpoint=https://ceph-s3.hostname.com s3 ls bucket_name
PRE indices/
PRE tests-Vf_GWcT0TsWuTNoLuf2mxg/
PRE tests-gIQqkXEiSo2AjQr87IJIPw/
PRE tests-ijVHsV_mTa-UNKUq2_k5wA/
2021-12-23 19:29:37 29 incompatible-snapshots
2021-12-23 19:29:25 177 index--1
2021-12-23 19:14:19 177 index-0
2021-12-23 19:14:20 8 index.latest
2021-12-23 19:14:03 27495 meta-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:12 27495 meta-dCAQNUiHQDeTPXjl3iUGHw.dat
2021-12-23 19:14:18 249 snap-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:30 249 snap-dCAQNUiHQDeTPXjl3iUGHw.dat
and a snapshot listing request:
# curl -X GET 'https://localhost:9200/_cat/snapshots/ceph?v'
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
20211223 SUCCESS 1640276043 16:14:03 1640276058 16:14:18 14.8s 1 5 0 5
why elasticsearch is requesting index --1 file in s3 repository?
according to my ingress logs the index --1 file did not try to upload to the s3 repository. I only see GET index --1 with 404 error
Is it ok that I will copy the file? there will be no problems when restoring a snapshot?