Elasticsearch Curator not respecting range on filtertype

Elastic Stack Elasticsearch
1

I'm trying to set up curator to alias and de-alias indices based on their age, but the removal isn't working as expected. If an older index doesn't exist, the alias is removed from the current index. This isn't a problem if I only want one index with the alias, but I'm looking at retaining 90 days of data within an alias. Does this mean I have to disable the removal until the required number of indices exist?

action: alias
description: "add new index to alias and remove oldest index if necessary"
options:
  name: "test"
  warn_if_no_indices: False
add:
  filters:
  - filtertype: pattern
    kind: prefix
    value: "test-"
    exclude:
  - filtertype: period
    source: name
    range_from: 0
    range_to: 0
    timestring: "%Y.%m.%d.%H"
    unit: "hours"
remove:
  filters:
  - filtertype: pattern
    kind: prefix
    value: "test-"
    exclude:
  - filtertype: period
    source: name
    range_from: -4
    range_to: -4
    timestring: "%Y.%m.%d.%H"
    unit: "hours"

The above config only works as expected if I have 4 previously existing indices in that alias. To me this is a bug, but maybe my config is wrong?

2

You have set warn_if_no_indices to False. Have you read what that option is for?

This setting specifies whether or not the alias action should continue with a warning or return immediately in the event that the filters in the add or remove section result in an empty index list.

This flag was added specifically to address this case, where the filters for add or remove might result in an empty list. But because the alias action is special, the ignore_empty_list option does not apply to either add or remove, as each is an independent index list object. To cover this possibility, the warn_if_no_indices option was added for the alias action, so that if that happened, adds or removes could proceed anyway.

It should make this work the way you were expecting.

3

Thanks Aaron that did fix it. I'm still not sure why though. To me the generated array of indices should never contain anything but what the conditions state, so the newest index that doesn't fit my parameters shouldn't even be seen by the dealias function. From what you're saying I can only assume that every index, irrespective of it's name, is part of the action list. Is that correct?

4

You are absolutely 100% correct. That's what happened. The filters resulted in no indices being seen, which is an empty list condition, which is an error to Curator. The warn_if_no_indices option tells the alias action to ignore an empty list condition for either the add or the remove, or both.

Kind of. When you initialize an "action" that uses indices, it creates an internal "IndexList" object, which at its initialization pulls all indices into it. The filters are then applied to this list, slowly winnowing out the ones you don't want to act on, leaving only the ones you do in the end. The alias action is special in that it creates not 1, but 2 "IndexList" objects: one for the add section, and one for the remove section. Each has its own filter section. Each can result in an empty list condition if there are no indices left in the List object after all filters have been applied.

Effectively, what happens in the alias action is that if you declared an add or remove section that resulted in an empty list condition, it will error out unless you set the warn_if_no_indices option to True, which tells Curator to basically remove that section from the body of the alias request, rather than have it be empty (which would result in a bad API call, otherwise).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.