How to backup all indices older than a date using Curator plugin?

Elastic Stack Elasticsearch
1

Hi,

It will be great if @theuntergeek can help me in this.

I have a curator action file like this;

actions:
  1:
    action: snapshot
    description: >-
    options:
      repository: mybackup
      name: myindexa-%Y%m%d%H%M%S
      ignore_unavailable: False
      include_global_state: True
      partial: False
      wait_for_completion: True
      skip_repo_fs_check: False
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: myindexa-
      exclude:
    - filtertype: age
      source: creation_date
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 5
      exclude:

I follow an index pattern having name followed by a hyphen and date like;

  • myindexa-2016-12-01
  • myindexb-2016-12-01
  • myindexc-2016-12-01
  • myindexd-2016-12-01
  • myindexe-2016-12-01
    etc

I want to take snapshot of all these indices at once. My current action file take snapshot of only one index (myindexa). How can I do this?

2

There are several ways to approach this, but it matters what you actually want in the snapshots.

Do you only want to snapshot indices with a year-month-day pattern in the name? Or are there some indices you want to snapshot that don't have that pattern? Because you have source: creation_date, Curator will ignore timestring: '%Y-%m-%d'. That line won't matter in your age filter.

If all of the indices you wanted to snapshot have a year-month-day pattern, you could simply exclude the prefix filter, and change to source: name.

    filters:
    - filtertype: age
      source: name
      timestring: '%Y-%m-%d'
      direction: older
      unit: days
      unit_count: 5

You could also filter by timestring pattern but not by the date indicated by the timestring in the index name (and still use the creation_date) this way:

    filters:
    - filtertype: pattern
      kind: timestring
      value: '%Y-%m-%d'
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 5

This filter block will only act on indices with a year-month-day pattern, and only those with a creation_date older than 5 days ago.

If you need to match certain patterns, you could also use a regex pattern kind:

    filters:
    - filtertype: pattern
      kind: regex
      value: '^(myindexa-|myindexb-|myindexc-|myindexn-).*$'
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 5

This filter block will only act on indices that begin with myindexa-, myindexb-, myindexc-, or myindexn-, and have a creation_date older than 5 days.

Hopefully this gives you some ideas. If you have more questions, feel free to ask!