S3 repository snapshot

Elastic Stack Elasticsearch
1

Hello,

I am trying to create a s3 repository snapshot that will take 1 exact day which is the day before 7 days, noting that all the indices I have is daily based, and this needs to be done every day at a specific time.

how this will be done since all the solutions are for taking periodical snapshots not for a specific day.

Thanks.

2

Snapshots works on entire indices, not on data inside indices.

How are your indices being created? Do you have daily indices?

3

Yes the indices are created on daily basis and I want to take a snapshot of 1 exact day which is the day before 7 days,

let's say I have the below indices.

logstash-2024.01.01
logstash-2024.01.02
logstash-2024.01.03
logstash-2024.01.04
logstash-2024.01.05
logstash-2024.01.06
logstash-2024.01.07

I want the policy to run every day at 00:00 and take the logstash-2024.01.01
then on the next day when logstash-2024.01.08 created the same thing the policy will take a snapshot of logstash-2024.01.02 and so on.

is it possible? And how is this done?

4

It is possible, but not using the native snapshot management UI in Kibana, you will need to build a custom tool, like a python script, that would create the snapshots in the way you want.

You can create a snapshot policy that would make a snapshot every day at a specific time, but you cannot specify any date math in the indices that would be snapshot, you need to use a index pattern, like logstash-* for example, so when this policy runs it will make a snaphost of all indices stating with logstash-*.

What is the logic behind waiting 7 days to make a snapshot? If for example your cluster has some failure you may lose data because you do not have snapshot of some indices.

5

the idea behind this logic is that we want to restore the snapshot on daily basis we dont want to make the restoration of all the indices once we want to restore each index with a specific day, it is not necessary to take the 7th day i just want to make each snapshot take one day.

what kind of python script do I have to make and how?

6

When restoring a snapshot you can specify the indices that you want to restore, you do not need to restore everything, it doesn´t matter if you have one or thousand indices in the same snapshot, you can restore just a specific index.

It would be way easier to create a snapshot policy that would look to your index pattern, like logstash-*.

This is up to you, you would need to use the Create Snapshot API and write something that would do what you want, but how you would do that is entirely up to you.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.