Snapshot AWS S3 bucket

Elastic Stack Elasticsearch
1

I have a script that does a snapshot to S3 and lately it has been failing (PARTIAL) and I have not idea
Here is the error, please help

"snapshot" : "snapshot_kibana_feb021606",
"version_id" : 1070199,
"version" : "1.7.1",
"indices" : [ ".marvel-kibana", "kibana-int", ".kibana" ],
"state" : "PARTIAL",
"start_time" : "2016-02-17T22:39:20.239Z",
"start_time_in_millis" : 1455748760239,
"end_time" : "2016-02-17T22:39:21.066Z",
"end_time_in_millis" : 1455748761066,
"duration_in_millis" : 827,
"failures" : [ {
"node_id" : "RY2PONaTSEu-xxxxxxx",
"index" : ".kibana",
"reason" : "IndexShardSnapshotFailedException[[.kibana][0] The operation is not valid for the object's storage class (Service: Amazon S3; Status Code: 403; Error Code: InvalidObjec
tState; Request ID: AFE9D8EDCE00A20F)]; nested: AmazonS3Exception[The operation is not valid for the object's storage class (Service: Amazon S3; Status Code: 403; Error Code: InvalidObje
ctState; Request ID: AFE9D8EDCE00A20F)]; ",
"shard_id" : 0,
"status" : "INTERNAL_SERVER_ERROR"

2

Please help
Thank you

3

Hi,

I'm not sure why this is occurring but you may want to take a look at this github issue which has a similar error:

You may want to check and see if prior snapshots of these indices exist in S3.

4

I do daily snapshot to S3 using the date format to make them unique
The backup has been running fine for the last 3 months it started failing for the past 2 weeks

tag=$(date '+%Y_%m_%d')
index="kibana"
curl -XPUT "localhost:9200/snapshot/s3_repository/snapshot${index}_${tag}?wait_for_completion=true" -d '{
"indices": ".marvel-kibana,kibana-int,.kibana",
"include_global_state": false }'

5

Do you use AWS?

In my case, In the past I used a low aws instance with low network and cpu. My snapshots become corrupted every day (I have index by day).

When I found this problem, I choose a bether instance and today I dont have failed snapshots.

if you use aws, see the monitoring graphs in your EC2 instance.

6

I think I may have find the problem (will be able to tell for sure in couple of days). We have lots of indices and my script backup them up at the end of the day after they have rollover. These 3 indices that are failing are the only indices that do not rollover, documents get added to them day by day. I have a policy on my S3 backup to move indices from S3 to IA (Infrequent Access) and to Glacier after 60 days.
It seems like those indices were moved to Glacier.
I am in the process of restoring them to S3 and run the snapshot again.
Will keep you posted.
Thanks

7

hmm, I had this problem.

the snapshot plugin not works with aws glacier. :slightly_smiling:

if you want to get a glacier snapshot you will have the same problem.

8

Hi Genesis..

Sorry to post question in your thread.
I have a doubt about this topic. If we create snapshots of elasticsearch cluster or node and store it in aws s3, how we can view the logs in kibana?

I mean for viewing logs in kibana, do we need to restore the snapshots back to elasticsearch.

I setup logserver in AWS EC2 and created a S3 bucket. Next I need to bind S3 with Elasticsearch ,but no idea of where to start.

Hope you will reply.

Thanks

9

Kibana reads what is indexed into elasticsearch.
So yes, you have to restore your data in elasticsearch if you want to be able to consume them into Kibana.

10

Okay. so these snapshots are using for backup only, right?

11

exactly.