Snapshot indices for last 7 days using Elasticsearch curator?

Elastic Stack Elasticsearch
1

HI,

How can i snapshot indices for past 7 days from today not older than that?

IS that possible in curator?

2

Of course! You can use the younger direction as a restriction:

---
actions:
  1:
    action: snapshot
    description: Snapshot the last 7 days of indices
    options:
        # ... options go here
    filters:
    - filtertype: age
      source: creation_date
      direction: younger
      unit: days
      unit_count: 8
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 1
    - filtertype: closed
    - filtertype: #... any additional filters

This should work, though if you're using source: name, then you'd need to change to add timestring and such.

It simply filters out indices that might be older than 7 days by requiring that indices also be younger than 8 days.

3

Thanks @theuntergeek

I need a little bit understanding on these younger and older.

LET the dates starts from data-2017-04-03 to 2017-04-11

What is the difference between these two scenarios:

Scenario 1 : I am giving younger filter in the first followed my older filter

   filters:
    - filtertype: pattern
      kind: prefix
      value: data-
      exclude:
    - filtertype: age
      source: name
      direction: younger
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 8
      exclude:
    filters:
    - filtertype: pattern
      kind: prefix
      value: data-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 1
      exclude:

My understanding: I dont know how this one works

Scenario 2: I am using older filter first followed my younger filter

filters:
    - filtertype: pattern
      kind: prefix
      value: data-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 1
      exclude:
    filters:
    - filtertype: pattern
      kind: prefix
      value: data-
      exclude:
    - filtertype: age
      source: name
      direction: younger
      timestring: '%Y-%m-%d'
      unit: days
      unit_count: 8
      exclude:

My understanding:

  1. First it will find the indices that are older than 1 day and in that it will take first 8

I know my understanding is wrong. Please correct me ...

FYI-For both the scenarios i am getting different responses.

Thanks

4

You have filters in there twice. I don't think that should work...

5

I converted the filter block I had above (plus using index- as the prefix for my tests) and this is what I made for command-line testing. You can see that I made indices for the 1st of April through today.

$ curator_cli show_indices --filter_list '[{"filtertype":"pattern", "kind":"prefix", "value":"index-"}]'
index-2017-04-01
index-2017-04-02
index-2017-04-03
index-2017-04-04
index-2017-04-05
index-2017-04-06
index-2017-04-07
index-2017-04-08
index-2017-04-09
index-2017-04-10
index-2017-04-11
index-2017-04-12

Now, when I add the older than 1 day and younger than 8 days age filters, I get this:

curator_cli show_indices --filter_list '[{"filtertype":"pattern", "kind":"prefix", "value":"index-"},{"filtertype":"age", "source":"name", "direction":"younger", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":8},{"filtertype":"age", "source":"name", "direction":"older", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":1}]'

And these were my results:

index-2017-04-05
index-2017-04-06
index-2017-04-07
index-2017-04-08
index-2017-04-09
index-2017-04-10
index-2017-04-11

Now, it doesn't matter what order the filters are in, the results will be the same:

$ curator_cli show_indices --filter_list '[{"filtertype":"age", "source":"name", "direction":"younger", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":8},{"filtertype":"pattern", "kind":"prefix", "value":"index-"},{"filtertype":"age", "source":"name", "direction":"older", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":1}]'

index-2017-04-05
index-2017-04-06
index-2017-04-07
index-2017-04-08
index-2017-04-09
index-2017-04-10
index-2017-04-11

curator_cli show_indices --filter_list '[{"filtertype":"age", "source":"name", "direction":"older", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":1},{"filtertype":"age", "source":"name", "direction":"younger", "timestring":"%Y-%m-%d", "unit":"days", "unit_count":8},{"filtertype":"pattern", "kind":"prefix", "value":"index-"}]'

index-2017-04-05
index-2017-04-06
index-2017-04-07
index-2017-04-08
index-2017-04-09
index-2017-04-10
index-2017-04-11

I would stick with this:

filters:
- filtertype: pattern
  kind: prefix
  value: data-
- filtertype: age
  source: name
  direction: older
  timestring: '%Y-%m-%d'
  unit: days
  unit_count: 1
- filtertype: age
  source: name
  direction: younger
  timestring: '%Y-%m-%d'
  unit: days
  unit_count: 8

Note that I removed the blank exclude: lines. They're not necessary. This is shorter and more readable (you had the data- prefix filter in twice).

No matter how you reorder these three filtertype blocks, the results will always be the same.

6

Thanks for your clear explanation @theuntergeek

 filters:
    - filtertype: age
      source: creation_date
      direction: younger
      unit: days
      unit_count: 8
    - filtertype: age
      source: creation_date
      direction: older
      unit: days
      unit_count: 1

The above code will find the indices that are older than 1 day and in that it will take first 7 days indices.(i.e.8-1=7) right.

Correct me if i am wrong
THANKS

7

Yes. You can clearly see that in the examples I pasted above. That was real output, not simulated. I created those indices (empty), and made the filters exactly as demonstrated.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.