Curator age filter not matcing all indices

Elastic Stack Elasticsearch
1

Hi,

We are creating daily indices and have a need of automatically closing them when they get older than 3 months. I have the following in my Curator action file:

14:
    action: close
    description: >-
      Close indices older than 3 months that are allocated to the archive-node
    options:
      delete_aliases: False
      disable_action: False
      ignore_empty_list: False
    filters:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y-%m-%d'
      unit: months
      unit_count: 3
    - filtertype: allocated
      key: box_type
      value: archive

This is the name of an example index that is not being closed:

logstash-azurestack-2019-02-28

{
  "logstash-azurestack-2019-02-28" : {
    "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "require" : {
              "box_type" : "archive"
            }
          }
        },
        "refresh_interval" : "30s",
        "number_of_shards" : "1",
        "provided_name" : "logstash-azurestack-2019-02-28",
        "creation_date" : "1551312020299",
        "number_of_replicas" : "0",
        "uuid" : "Kg0gYT7dTSylxlyupV1r3w",
        "version" : {
          "created" : "6030299"
        }
      }
    }
  }
}

How come the index isn't getting closed?

2

Hi @victor.nilsson,

What is the o/p of curator logs showing. could you please post it for analysis?

Regards,
Harsh Bajaj

3

Sure.

2019-06-23 20:18:42,719 INFO Preparing Action ID: 14, "close"
2019-06-23 20:18:42,722 INFO Trying Action ID: 14, "close": Close indices older than 3 months that are allocated to the archive-node
2019-06-23 20:19:13,755 INFO Skipping action "close" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-06-23 20:19:13,755 INFO Action ID: 14, "close" completed.
2019-06-23 20:19:13,755 INFO Job completed.

4

Hi @victor.nilsson,

Could you please enable the Debug mode in curator.yml file and paste the debug logs instead of INFO?

It will help me to understand where is the problem.

Regards,
Harsh Bajaj

5

Sure thing. The output is pretty damn large, is there a way to share it to you?

EDIT:

2019-06-24 11:50:55,287 DEBUG          curator.indexlist       __not_actionable:39   Index logstash-iazurestack-2019-02-28 is not actionable, removing from list.
2019-06-24 11:50:55,287 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: logstash-iazurestack-2019-02-28: Routing (mis)match: index.routing.allocation.require.box_type=archive.

I can see that it mismatches on the routing allocation awareness. However, the index is on the archive-node as mentioned in the OP.

6

Could you please try with dry run without using below section and check it is showing in actionable list or not.

filtertype: allocated
key: box_type
value: archive

if its working then please try below configuration for archive. You need to edit "options" section and add below parameters.

options:
  key: box_type
  value: archive
  allocation_type: require
  wait_for_completion: True
filters:

Regards
Harsh Bajaj

7

@victor.nilsson,

You have not define any pattern in your action file. So you should define the pattern and your action file will be look like below:

actions:
  1:
    action: close
    description: >-
      Close indices older than 30 days (based on index name), for logstash-
      prefixed indices.
    options:
      delete_aliases: False
      timeout_override:
      continue_if_exception: False
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-azurestack-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 90
      exclude:

Please try above action file. If you face any issue let me know.

Thanks.

8

This did not work. I got the same errors as before.

9

Hi, Thanks but you're missing the crucial feature that we need. We only want to close indices that are allocated to our archive node.

- filtertype: allocated
      key: box_type
      value: archive

I tried your configfile with the allocated filtertype added and still got the same error.

10

I solved it by adding "exclude: false" to the allocated filter.

  • filtertype: allocated
    key: box_type
    value: archive
    exclude: false

Thanks all for the help

11

Yes,i shared the some properties which you can add/remove as per your requirement.

Great if it works!!

12

@victor.nilsson,

Great..yes by default exclude is true in filtertype allocated. I didn't check it earlier.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.