Backup to s3 repo from elasticsearch 6.8.22

Elastic Stack Elasticsearch
1

hi
I am using elasticsearch 6.8.22 with plugin repository-s3
I also use ceph s3 storage running on rook(k8s)

I'm trying to set up a snapshot backup for it:

I create a repository command:

 curl -sk -X PUT "https://localhost:9200/_snapshot/ceph?pretty" -H 'Content-Type: application/json' -d'
 {
   "type": "s3",
   "settings": {
     "bucket": "bucket_name",
     "endpoint": "ceph-s3.hostname.com:443",
     "protocol": "https",
     "region": "region_name",
     "base_path": "bucket_name",
     "read_timeout": "3600s",
     "disable_chunked_encoding": true
   }
 }'

answer:

 {
   "acknowledged" : true
 }

Everything went well and I can start backup:

 now=`date +%Y%m%d`
 curl -k -X PUT 'https://localhost:9200/_snapshot/ceph/'"$now"'?wait_for_completion=true&pretty'   -H 'Content-Type: application/json'  -d'  {
   "indices": "bases_2"
 }'

answer:

 {
   "snapshot" : {
     "snapshot" : "20211223",
     "uuid" : "8SMS5XFKQtiSW1jKsjEyzA",
     "version_id" : 6082299,
     "version" : "6.8.22",
     "indices" : [
       "bases_2"
     ],
     "include_global_state" : true,
     "state" : "SUCCESS",
     "start_time" : "2021-12-23T16:14:03.709Z",
     "start_time_in_millis" : 1640276043709,
     "end_time" : "2021-12-23T16:14:18.560Z",
     "end_time_in_millis" : 1640276058560,
     "duration_in_millis" : 14851,
     "failures" : [ ],
     "shards" : {
       "total" : 5,
       "failed" : 0,
       "successful" : 5
     }
   }
 }

In the master node logs:

[2021-12-23T19:10:13,759][INFO ][o.e.s.SnapshotsService ] [node-01] snapshot [ceph:20211223/dCAQNUiHQDeTPXjl3iUGHw] started
[2021-12-23T19:10:32,625][INFO ][o.e.s.SnapshotsService ] [node-01] snapshot [ceph:20211223/dCAQNUiHQDeTPXjl3iUGHw] completed with state [SUCCESS]

But I cannot see the created snapshot in the snapshot listing

 curl -X GET 'https://localhost:9200/_cat/snapshots/ceph?v'
 id status start_epoch start_time end_epoch end_time duration indices successful_shards ailed_shards total_shards

 ..empty output..

The state of s3 repository immediately after the completion of the snapshot:

 $ aws --profile=ceph --endpoint=https://ceph-s3.hostname.com s3 ls bucket_name/
                             PRE indices/
                            PRE tests-Vf_GWcT0TsWuTNoLuf2mxg/
                            PRE tests-gIQqkXEiSo2AjQr87IJIPw/
                            PRE tests-ijVHsV_mTa-UNKUq2_k5wA/
 2021-12-23 19:14:19        177 index-0
 2021-12-23 19:14:20          8 index.latest
 2021-12-23 19:14:03      27495 meta-8SMS5XFKQtiSW1jKsjEyzA.dat
 2021-12-23 19:10:12      27495 meta-dCAQNUiHQDeTPXjl3iUGHw.dat
 2021-12-23 19:14:18        249 snap-8SMS5XFKQtiSW1jKsjEyzA.dat
 2021-12-23 19:10:30        249 snap-dCAQNUiHQDeTPXjl3iUGHw.dat

In the logs of s3 ceph (ingress) access when requesting a snapshot listing, I see a file access:

{
  "protocol": "HTTP/1.1",
  "path": "/bucket_name/index--1",
  "bytes_sent": 213,
  "bytes_received": 0,
  "user_agent": "aws-sdk-java/1.11.406 Linux/5.4.0-65-generic OpenJDK_64-Bit_Server_VM/25.282-b08 java/1.8.0_282",
  "method": "GET",
  "upstream_cluster": "outbound|80||rook-ceph-rgw-gib.rook-ceph.svc.cluster.local",
  "response_code_details": "via_upstream",
  "upstream_transport_failure_reason": null,
  "response_code": 404,
  "referer": null
}

(which is not on my server) (!)

I can see the index-0 file in the s3 ceph repository.
I copy it to the index - 1 file:

 aws --profile=ceph --endpoint=https://ceph-s3.hostname.com s3 ls bucket_name
                           PRE indices/
                           PRE tests-Vf_GWcT0TsWuTNoLuf2mxg/
                           PRE tests-gIQqkXEiSo2AjQr87IJIPw/
                           PRE tests-ijVHsV_mTa-UNKUq2_k5wA/
2021-12-23 19:29:37         29 incompatible-snapshots
2021-12-23 19:29:25        177 index--1
2021-12-23 19:14:19        177 index-0
2021-12-23 19:14:20          8 index.latest
2021-12-23 19:14:03      27495 meta-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:12      27495 meta-dCAQNUiHQDeTPXjl3iUGHw.dat
2021-12-23 19:14:18        249 snap-8SMS5XFKQtiSW1jKsjEyzA.dat
2021-12-23 19:10:30        249 snap-dCAQNUiHQDeTPXjl3iUGHw.dat

and a snapshot listing request:

# curl -X GET 'https://localhost:9200/_cat/snapshots/ceph?v'
id        status start_epoch start_time end_epoch  end_time duration indices successful_shards failed_shards total_shards
20211223 SUCCESS 1640276043  16:14:03   1640276058 16:14:18    14.8s       1                 5             0            5

why elasticsearch is requesting index --1 file in s3 repository?
according to my ingress logs the index --1 file did not try to upload to the s3 repository. I only see GET index --1 with 404 error

Is it ok that I will copy the file? there will be no problems when restoring a snapshot?

2

Not really no. The warning in the 7.16 manual also applies to older versions:

Don’t modify anything within the repository or run processes that might interfere with its contents. If something other than Elasticsearch modifies the contents of the repository then future snapshot or restore operations may fail, reporting corruption or other data inconsistencies, or may appear to succeed having silently lost some of your data.

However I don't know why the snapshot is not shown in the listing. Does this only happen when using Ceph or does it also happen when using the real S3?

3

does it also happen when using the real S3?

I have not tested with real s3. = ((

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.