First attempt - rollover using filters - extra keys not allowed @ data['filters']

Elastic Stack Elasticsearch
1

curator, version 5.6.0

rollover.yaml:

---

actions:
  1:
    action: rollover
    description: >-
      Rollover the index associated with alias 'aliasname', which should be in the
      format of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
    options:
      disable_action: False
      name: nginx_logs
      warn_if_no_indices: True
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: period
      period_type: absolute
      source: name
      timestring: '%Y.%m.%d'
      unit: months
      date_from: 2018.01
      date_from_format: '%Y.%m'
      date_to: 2019.01
      date_to_format: '%Y.%m'

running:

curator --dry-run rollover.yaml

returns:

curator.exceptions.ConfigurationError: Configuration: structure: Location: Action ID "1": Bad Value: "[{'kind': 'prefix', 'filtertype': 'pattern', 'value': 'filebeat-'}, {'date_from': 2018.01, 'filtertype': 'period', 'date_to_format': '%Y.%m', 'source': 'name', 'period_type': 'absolute', 'timestring': '%Y.%m.%d', 'date_to': 2019.01, 'date_from_format': '%Y.%m', 'unit': 'months'}]", extra keys not allowed @ data['filters']. Check configuration file.

I pretty much took it all from the examples in the documentation.
Is this a yaml formatting issue or something else?

2

Hi @jroberts235,

This is not the formatting issue in yaml file. It seems there is some problem in your filter section.

Regards,

3

Thanks for the response.

I agree, it's not a formatting issue.
I don't see anything in the docs to confirm that the Rollover action supports a filter. That said, nothing saying that it doesn't. It seems like a logical pairing of action and filter type...

But whenever a filter is added, it throws the error regardless if the name field is populated in the action or not, which I thought might be the issue.

To narrow the problem down, I've simplified the action to this:

actions:
  1:
    action: rollover
    options:
      disable_action: False
      #name: filebeat
      #warn_if_no_indices: True
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
      - filtertype: pattern
        kind: prefix
        value: filebeat-
    #- filtertype: period
    #  period_type: absolute
    #  source: name
    #  timestring: '%Y.%m.%d'
    #  unit: months
    #  date_from: 2018.01
    #  date_from_format: '%Y.%m'
    #  date_to: 2019.01
    #  date_to_format: '%Y.%m'

And still get the same error.

4

something is broken. I get the same error "extra keys not allowed" when I run this example taken straight from the docs:

description: >-
  Rollover the index associated with index 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False```


Am I the only one seeing this behavior??  Is anyone else completely annoyed with the level of error messaging that comes out of Curator??
5

This is my current script:

---
actions:
  1:
    action: rollover
    options:
      disable_action: False
      name: filebeat
      conditions:
      #warn_if_no_indices: True
      #new_index: "<prefix-{now/d}-1>"
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
    - filtertype: pattern
      kind: prefix
      value: '^file*'
    - filtertype: age
      source: creation_date
      direction: older
      unit: months
      unit_count: 3

and this is the current error:

> curator --dry-run actions/rollover.yaml
Traceback (most recent call last):
  File "/Users/jroberts235/Library/Python/2.7/bin/curator", line 11, in <module>
    sys.exit(cli())
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/cli.py", line 213, in cli
    run(config, action_file, dry_run)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/cli.py", line 119, in run
    action_dict = validate_actions(action_config)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/utils.py", line 1410, in validate_actions
    loc
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/validators/schemacheck.py", line 69,in result
    self.test_what, self.location, self.badvalue, self.error)
curator.exceptions.ConfigurationError: Configuration: structure: Location: Action ID "1": Bad Value: "[{'kind': 'prefix', 'filtertype': 'pattern', 'value': '^file*'}, {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'months'}]", extra keys not allowed @ data['filters']. Check configuration file.

I'm ready to thrown in the towel and look for another solution.

6

Rollover doesn’t use filters because the name is an alias. There currently is no way to filter aliases, so Curator can only rollover one alias at a time.

A feature request to allow multiple aliases to be specified is here

7

With that said, and me home now, I am taking a closer look:

This won't work because there are no conditions. You need to specify at least one of max_size, max_docs, or max_age.

And this isn't working if it's missing the action clause:

description: >-
  Rollover the index associated with index 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False

A functional merging of the two would be:

--- 
actions:
  1:
    action: rollover
    options:
      name: filebeat
      conditions:
        max_age: 5d
      extra_settings: 
        index.number_of_shards: 3
        index.number_of_replicas: 1

This would rollover at 5 days.

I'm sorry that Curator can't accurately parse YAML structural errors in a pleasing way.

extra keys not allowed @ data['filters']

is YAML linting code for, "There shouldn't be a filters key here."

8

Thank you for your response.

The "action" that I posted that is in fact missing the "action" statement, is just a cut and paste error from when I posted it. The actual script does indeed include the "action" line, and when run, I do get the error I posted above.

9

Have you tried the block I shared? If so, what are the error(s)?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.