Snapshot - Repository S3 (Strategy)

Elastic Stack Elasticsearch
1

Hi!

I'm doing a backup in our S3. So far we're using a monthly index, but we're planning to do a daily backup using curator.

Versions:

  • Elasticsearch 5.6.1
  • Curator 5.2

Here are some questions:

  1. Is it good to do a daily snapshot for each index ?
  2. Using the curator filters, is it possible to grab the current date? because when using timestring and regex, it still grabs all the similar pattern which is index-YEAR-MONTH.
  3. For snapshot, is it okay to include old indices even it will be deleted since I'm going to create snapshot based from creation time
  4. Since we're using a monthly index, we expect to create a snapshot while there's an on-going index. Based from the docs, this shouldn't be an issue.

Thanks!

2
  1. Is it good to do a daily snapshot for each index ?

On cloud.elastic.co we are doing snapshot every 30 minutes. Everyday is fine.

  1. Using the curator filters, is it possible to grab the current date? because when using timestring and regex, it still grabs all the similar pattern which is index-YEAR-MONTH.

I don't know. @theuntergeek probably can help.

  1. For snapshot, is it okay to include old indices even it will be deleted since I'm going to create snapshot based from creation time

Not sure I understood.

  1. Since we're using a monthly index, we expect to create a snapshot while there's an on-going index. Based from the docs, this shouldn't be an issue.

Not an issue indeed. Only data available at the moment you start the snapshot will be saved though.

3

Awesome! Thanks for answering the questions.

For number 3, Let's say I have the following indices:

  • index-2017-07
  • index-2017-08
  • index-2017-09
  • index-2017-10

Since I'm planning to create a daily snapshot it will look like this:

"snapshot" : "index-backup-2017-10-09",
  "uuid" : "xxxxxxxxxxxxxxxxxx",
  "version_id" : xxxxxxx,
  "version" : "5.6.1",
  "indices" : [
    "index-2017-09",
    "index-2017-10",
    "index-2017-07",
    "index-2017-08"
  ],

Since I only need around 3 months worth of data, I plan to delete indices which are older than 3 months. Let's say it's already November, the index-2017-07 will be deleted. Will that affect the snapshot created when index-2017-07 is present? Based from the docs, it shouldn't since it's incremental.

4

I don't understand the question. Can you please rephrase this?

To understand better how snapshots work, I recommend reading this comment on GitHub.

5

For the curator part, let's say I have index named "filebeat-%Y-%m". So far, I have indices for filebeat starting June which results to indices below:

  • filebeat-2017-06
  • filebeat-2017-07
  • filebeat-2017-08
  • filebeat-2017-09
  • filebeat-2017-10

Right now it's October, Is there a way in curator to just grab the current month which is October and do a snapshot of it? or that's a not good thing since if I'm doing a backup daily then it doesn't matter if I include all filebeat-%Y-%m indices since it's incremental.

6

Yes. See the period filter

---
actions:
  1:
    action: snapshot
    # snapshot settings and options go here
    # ...
    filters:
      - filtertype: pattern
        kind: prefix
        value: filebeat-
      - filtertype: period
        source: name
        range_from: 0
        range_to: 0
        timestring: '%Y.%m'
        unit: months

This example will select the current month's filebeat index (which are presumed to be named with a monthly timestamp) and perform a snapshot of it.

1 Like
7

Awesome! It looks like I missed out the period filter in the docs.
Thanks for bringing that one!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.