How to take Manual backup of AWS elasticsearch to s3 bucket with indexname

Elastic Stack Elasticsearch
1

Hi I am trying to take back up of my AWS elasticsearch indices . and I configured IAM role and policy and using curl to take the back . All the indices are backed up in s3 bucket. But inside the indices folder in s3 bucket new folder is created with 0RZdYteiQLCbvGJacpC5IA name not the index name used in elastic search. I am not able find which index back up . How to take the back up with index name.

For ex : my aws elasticsearch index name :alb-accesslog-2018.11.28
folder inside the s3 bucket is like 0RZdYteiQLCbvGJacpC5IA

2

You can call the _cat/indices API to get the index id.

BUT you should use snapshot and restore feature instead of going to the hard disk manually...

3

i am taking the snapshot for manual backup inside s3 ,but inside the s3 bucket i am not getting snapshot with indices name. its something other than indices number.

i am using this command to take backup

curl -XPUT 'https://search-mb-production-app-bhbhhbh.us-west-2.es.amazonaws.com/_snapshot/my-snapshot-repo/28-11-2018'

and getting output in s3 bucket like
https://s3.console.aws.amazon.com/s3/buckets/elasticsearch-backup-indices/indices/ 0RZdYteiQLCbvGJacpC5IA

i want like
https://s3.console.aws.amazon.com/s3/buckets/elasticsearch-backup-indices/indices/alb-access.log-2018-11-28

access.log-2018-11-28 in index name in Elasticsearch

4

But why do you want to manually access to those directories ?

BTW did you look at https://www.elastic.co/cloud and https://aws.amazon.com/marketplace/pp/B01N6YCISK ?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL and what is coming like Canvas...

5

I want to know which folder backup is for which index.

I have 17 index in aws elastic search 17 folder are created in s3 with different name. I dont know which folder belong to which index.

6

_cat/indices

could please advice the full command

7

I am following below link for "Use Amazon S3 to Store a Single Amazon Elasticsearch Service Index"https://aws.amazon.com/blogs/database/use-amazon-s3-to-store-a-single-amazon-elasticsearch-service-index/

I am successfully able to upload the indices from AWS Elasticseach to S3 bucket, but also noticed that folder name has changed on S3 bucket.

Folder name on ElasticSearch: enter image description here

Folder name on S3 bucket: enter image description here

8

https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-indices.html

9

thank you

10

_cat-indices we will get the UUID allocated for index . but s3 bucket name are different
index : alb-accesslog-2018.11.24
uuid : z6R3py0mQNWFC0y6_oIULw

in s3 bucket folder name is 4hIH1pCTQlG5rFkatJA7fg

11

I want to know how to relate s3 folder name with aws elastic search index name?

because I have 16 index in AWS elastic search and 16 folder are there inside the indices folder in s3 bucket. I want to know which index snapshot is stored in which folder in s3?

13

Hi dadoonet

could please advice how to take back up of single index. I used the command like
curl -XPUT 'http://search-mb-production-app-ssssssss.us-west-2.es.amazonaws.com/_snapshot/my-snapshot-repo/alb-accesslog-2018.11.29'
-d'{
"indices": "alb-accesslog-2018.11.29",
"ignore_unavailable": true,
"include_global_state": false
}'
but got response like
{"accepted":true}./sample.sh: line 6: $'-d{\n "indices": "alb-accesslog-2018.11.29",\n "ignore_unavailable": true,\n "include_global_state": false\n }': command not found
and when i check the status all the indices are backed up

14

Your shell script is probably wrong. Try with a space may be between -d and ` ?

Like:

curl -XPUT 'http://search-mb-production-app-tissfibb4qog2kiscfoghb7lfe.us-west-2.es.amazonaws.com/_snapshot/my-snapshot-repo/alb-accesslog-2018.11.29' -d '{
  "indices": "alb-accesslog-2018.11.29",
  "ignore_unavailable": true,
  "include_global_state": false
}'
15

BTW are you sure you really want to have your cluster exposed to internet?

16

nope

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.