Curator - Skipping action "alias" due to empty list: <class 'curator.exceptions.NoIndices

pbabik · February 13, 2019, 4:52pm

Hello,

after we upgraded to 6.2.4 we are having really odd issue only on 1 production cluster. I'm getting empty list of indices where there are clearly indices which suits the condition. Whats more it works on test environment in the same setup.
If I don't use "ignore empty list" option it does throw an error. For filtertype pattern i have used prefix and regex, both the same outcome. Action.yml was copied from test environment where it works so I avoid any typos, still the same. Curator version is 5.5.4. Any ideas pls? Thx

Works:

5:
    action: alias
    description: "Add/Remove selected indices to or from the specified alias"
    options:
      name: analytics
      warn_if_no_indices: False
      ignore_empty_list: True
      continue_if_exception: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: regex
        value: '^analytics'
      - filtertype: age
        source: name
        direction: younger
        timestring: '%Y.%W'
        unit: weeks
        unit_count: 2
        exclude:
    remove:
      filters:
      - filtertype: pattern
        kind: regex
        value: '^analytics'
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y.%W'
        unit: weeks
        unit_count: 2
        exclude:

_cat/indices/analytics*

green open analytics-2019.06w-000001 0OQfVY5vRXihtIUOEco1hA 4 1 558191086         0 325.3gb 162.6gb
green open analytics-2019.06w-000002 qXJyu__tTISNrjlItksQCw 4 1 199601308         0 109.6gb  54.7gb
green open analytics-2019.07w-000001 gTv6ksskRj20pOGJZJT9wQ 4 1 433071540         0 258.3gb 130.4gb

log:

2019-02-13 17:04:48,581 INFO      Trying Action ID: 5, "alias": Add/Remove selected indices to or from the specified alias
2019-02-13 17:04:50,246 INFO      Updating aliases...
2019-02-13 17:04:50,246 INFO      Alias actions: {'actions': [{'add': {'index': 'analytics-2019.05w-000001', 'alias': 'analytics'}}, {'add': {'index': 'analytics-2019.07w-000001', 'alias': 'analytics'}}, {'add': {'index': 'analytics-2019.05w-000002', 'alias': 'analytics'}}, {'add': {'index': 'analytics-2019.06w-000002', 'alias': 'analytics'}}, {'add': {'index': 'analytics-2019.06w-000001', 'alias': 'analytics'}}]}
2019-02-13 17:04:50,268 INFO      Action ID: 5, "alias" completed.

Doesn't work:

6:
    action: alias
    description: "Add/Remove selected indices to or from the specified alias"
    options:
      name: audit
      warn_if_no_indices: False
      ignore_empty_list: True
      continue_if_exception: True
      disable_action: False
    add:
      filters:
      - filtertype: pattern
        kind: prefix
        value: audit-
      - filtertype: age
        source: name
        direction: younger
        timestring: '%Y.%m'
        unit: months
        unit_count: 2
        exclude:
    remove:
      filters:
      - filtertype: pattern
        kind: prefix
        value: audit-
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y.%m'
        unit: months
        unit_count: 2
        exclude:

_cat/indices/audit*

green open audit-2018    I5Rr-1I_S5iF-k14QBWgmw 4 1 181787694 135286044 177.5gb 88.7gb
green open audit-2019.02 yTDMLejxQJWtBA9ttbwpJw 4 1   8643326         0  16.6gb  8.3gb
green open audit-2019.01 hPMhiPSBRFCLsIy0Vn09Lw 4 1  30760434       100    42gb   21gb

logs:

2019-02-13 17:04:50,268 INFO      Preparing Action ID: 6, "alias"
2019-02-13 17:04:50,319 INFO      Trying Action ID: 6, "alias": Add/Remove selected indices to or from the specified alias
2019-02-13 17:04:50,667 INFO      Skipping action "alias" due to empty list: <class 'curator.exceptions.NoIndices'>
2019-02-13 17:04:50,668 INFO      Action ID: 6, "alias" completed.
2019-02-13 17:04:50,668 INFO      Job completed.

theuntergeek · February 13, 2019, 5:07pm

Please reformat the preformatted bits between triple back ticks:

```
PASTE HERE
```

It's too hard to read what you intend here otherwise.

pbabik · February 13, 2019, 5:10pm

Thanks I didn't know how to do that

theuntergeek · February 13, 2019, 5:22pm

You say "after you upgraded to 6.2.4" these problems started. From which version did you upgrade?

pbabik · February 13, 2019, 5:27pm

Upgraded from 5.5.2, but the weird part is that i have same indices in test env and there it works.

2019-02-13 18:01:04,727 INFO      Preparing Action ID: 6, "alias"
2019-02-13 18:01:04,798 INFO      Trying Action ID: 6, "alias": Add/Remove selected indices to or from the specified alias
2019-02-13 18:01:05,473 INFO      Updating aliases...
2019-02-13 18:01:05,473 INFO      Alias actions: {'actions': [{'add': {'index': 'audit-2019.01', 'alias': 'audit'}}, {'add': {'index': 'audit-2019.02', 'alias': 'audit'}}]}
2019-02-13 18:01:05,483 INFO      Action ID: 6, "alias" completed.

theuntergeek · February 13, 2019, 5:41pm

If you dial up the loglevel to DEBUG, it should show how it calculates index ages in the filters, and why an index is or is not selected. That will help explain what's going on here.

pbabik · February 15, 2019, 11:15am

I have the log but its too big to paste it here. The message is correct because in action id 6 , the REMOVE part is evaluated and curator correctly removes all indices from the list.. But the ADDing part is not evaluated at all and the process.action 78 and 79 are not executed.

grep process_action curator.2019-02-13_18_44.log

2019-02-13 18:44:19,101 DEBUG                curator.cli         process_action:44   Configuration dictionary: {'action': 'alias', 'description': 'Add/Remove selected indices to or from the specified alias', 'options': {'name': 'analytics', 'warn_if_no_indices': False, 'extra_settings': {}}, 'add': {'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^analytics', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'younger', 'timestring': '%Y.%W', 'unit': 'weeks', 'unit_count': 2, 'exclude': False, 'stats_result': 'min_value', 'epoch': None}]}, 'remove': {'filters': [{'filtertype': 'pattern', 'kind': 'regex', 'value': '^analytics', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%W', 'unit': 'weeks', 'unit_count': 2, 'exclude': False, 'stats_result': 'min_value', 'epoch': None}]}}
2019-02-13 18:44:19,101 DEBUG                curator.cli         process_action:45   kwargs: {'master_timeout': 30, 'dry_run': False}
2019-02-13 18:44:19,101 DEBUG                curator.cli         process_action:50   opts: {'name': 'analytics', 'warn_if_no_indices': False, 'extra_settings': {}}
2019-02-13 18:44:19,101 DEBUG                curator.cli         process_action:62   Action kwargs: {'name': 'analytics', 'warn_if_no_indices': False, 'extra_settings': {}}
2019-02-13 18:44:19,101 DEBUG                curator.cli         process_action:67   Running "ALIAS" action
2019-02-13 18:44:20,870 DEBUG                curator.cli         process_action:73   Removing indices from alias "analytics"
2019-02-13 18:44:20,904 DEBUG                curator.cli         process_action:78   Adding indices to alias "analytics"
2019-02-13 18:44:20,935 DEBUG                curator.cli         process_action:99   Doing the action here.
2019-02-13 18:44:21,002 DEBUG                curator.cli         process_action:44   Configuration dictionary: {'action': 'alias', 'description': 'Add/Remove selected indices to or from the specified alias', 'options': {'name': 'audit', 'warn_if_no_indices': False, 'extra_settings': {}}, 'add': {'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'audit-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'younger', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 2, 'exclude': False, 'stats_result': 'min_value', 'epoch': None}]}, 'remove': {'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'audit-', 'exclude': False}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m', 'unit': 'months', 'unit_count': 2, 'exclude': False, 'stats_result': 'min_value', 'epoch': None}]}}
2019-02-13 18:44:21,002 DEBUG                curator.cli         process_action:45   kwargs: {'master_timeout': 30, 'dry_run': False}
2019-02-13 18:44:21,002 DEBUG                curator.cli         process_action:50   opts: {'name': 'audit', 'warn_if_no_indices': False, 'extra_settings': {}}
2019-02-13 18:44:21,003 DEBUG                curator.cli         process_action:62   Action kwargs: {'name': 'audit', 'warn_if_no_indices': False, 'extra_settings': {}}
2019-02-13 18:44:21,003 DEBUG                curator.cli         process_action:67   Running "ALIAS" action
2019-02-13 18:44:22,882 DEBUG                curator.cli         process_action:73   Removing indices from alias "audit"

theuntergeek · February 15, 2019, 2:39pm

I appreciate that you're not seeing something finish, but I need to see why, and the grep will hide that. You can paste the relevant bits into a pastebin or GitHub gist and put the link here.

pbabik · February 15, 2019, 3:16pm

Logs from action id 5 & 6 can be seen here: https://pastebin.com/SMHdRvrQ

theuntergeek · February 15, 2019, 3:46pm

Okay, here's why you need warn_if_no_indices: true in your settings. It won't even evaluate the add portion if there are no indices in the remove block, or vice versa.

2019-02-13 18:44:22,901 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "audit-2019.02" age (1548979200), direction: "older", point of reference, (1544895862)
2019-02-13 18:44:22,901 DEBUG          curator.indexlist       __not_actionable:39   Index audit-2019.01 is not actionable, removing from list.
2019-02-13 18:44:22,901 DEBUG          curator.indexlist            __excludify:58   Removed from actionable list: Index "audit-2019.01" age (1546300800), direction: "older", point of reference, (1544895862)
2019-02-13 18:44:22,901 DEBUG          curator.indexlist        iterate_filters:1184 Post-instance: []
2019-02-13 18:44:22,901 DEBUG              curator.utils             isdatemath:1879 opener =  a, closer = t
2019-02-13 18:44:22,901 DEBUG          curator.indexlist       empty_list_check:224  Checking for empty list
2019-02-13 18:44:22,901 INFO                 curator.cli                    run:175  Skipping action "alias" due to empty list: <class 'curator.exceptions.NoIndices'>

The problem here is that actions in Curator are designed to catch empty list conditions and exit. However, Aliases are special, and have both add and remove filters, either or both of which might result in an empty list condition. A special work-around was added for this very use case, and it is the warn_if_no_indices option. See this thread for more detailed information on what lead to that option being added.

pbabik · February 15, 2019, 5:04pm

Amazing, it worked!! I would have never thought of this. Thanks!

system · March 15, 2019, 5:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.