# First attempt - rollover using filters - extra keys not allowed @ data\['filters'\]

**URL:** <https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454>\
**Category:** Elasticsearch\
**Created:** [January 9, 2019, 4:52am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454 "2019-01-09T04:52:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![jroberts235](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@jroberts235](https://discuss.elastic.co/u/jroberts235)\
**Post date:** [January 9, 2019, 4:52am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/1 "2019-01-09T04:52:16Z")

</div>

`curator, version 5.6.0`

rollover.yaml:

```auto
---

actions:
  1:
    action: rollover
    description: >-
      Rollover the index associated with alias 'aliasname', which should be in the
      format of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
    options:
      disable_action: False
      name: nginx_logs
      warn_if_no_indices: True
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: period
      period_type: absolute
      source: name
      timestring: '%Y.%m.%d'
      unit: months
      date_from: 2018.01
      date_from_format: '%Y.%m'
      date_to: 2019.01
      date_to_format: '%Y.%m'

```

running:

`curator --dry-run rollover.yaml`

returns:

`curator.exceptions.ConfigurationError: Configuration: structure: Location: Action ID "1": Bad Value: "[{'kind': 'prefix', 'filtertype': 'pattern', 'value': 'filebeat-'}, {'date_from': 2018.01, 'filtertype': 'period', 'date_to_format': '%Y.%m', 'source': 'name', 'period_type': 'absolute', 'timestring': '%Y.%m.%d', 'date_to': 2019.01, 'date_from_format': '%Y.%m', 'unit': 'months'}]", extra keys not allowed @ data['filters']. Check configuration file.`

I pretty much took it all from the examples in the documentation.  
Is this a yaml formatting issue or something else?

---

<div class="post-metadata">

**Author:** ![harshbajaj16](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harshbajaj16/32/44970_2.png) [@harshbajaj16](https://discuss.elastic.co/u/harshbajaj16)\
**Post date:** [January 9, 2019, 11:01am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/2 "2019-01-09T11:01:03Z")

</div>

Hi @jroberts235,

This is not the formatting issue in yaml file. It seems there is some problem in your filter section.

Regards,

---

<div class="post-metadata">

**Author:** ![jroberts235](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@jroberts235](https://discuss.elastic.co/u/jroberts235)\
**Post date:** [January 9, 2019, 4:52pm UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/3 "2019-01-09T16:52:38Z")

</div>

Thanks for the response.

I agree, it's not a formatting issue.  
I don't see anything in the docs to confirm that the Rollover action supports a filter. That said, nothing saying that it doesn't. It seems like a logical pairing of action and filter type...

But whenever a filter is added, it throws the error regardless if the `name` field is populated in the action or not, which I thought might be the issue.

To narrow the problem down, I've simplified the action to this:

```auto
actions:
  1:
    action: rollover
    options:
      disable_action: False
      #name: filebeat
      #warn_if_no_indices: True
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
      - filtertype: pattern
        kind: prefix
        value: filebeat-
    #- filtertype: period
    # period_type: absolute
    # source: name
    # timestring: '%Y.%m.%d'
    # unit: months
    # date_from: 2018.01
    # date_from_format: '%Y.%m'
    # date_to: 2019.01
    # date_to_format: '%Y.%m'

```

And still get the same error.

---

<div class="post-metadata">

**Author:** ![jroberts235](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@jroberts235](https://discuss.elastic.co/u/jroberts235)\
**Post date:** [January 16, 2019, 2:56am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/4 "2019-01-16T02:56:27Z")

</div>

something is broken. I get the same error "extra keys not allowed" when I run this example taken straight from the docs:

````auto
description: >-
  Rollover the index associated with index 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False```

Am I the only one seeing this behavior?? Is anyone else completely annoyed with the level of error messaging that comes out of Curator??
````

---

<div class="post-metadata">

**Author:** ![jroberts235](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@jroberts235](https://discuss.elastic.co/u/jroberts235)\
**Post date:** [January 16, 2019, 2:58am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/5 "2019-01-16T02:58:19Z")

</div>

This is my current script:

```auto
---
actions:
  1:
    action: rollover
    options:
      disable_action: False
      name: filebeat
      conditions:
      #warn_if_no_indices: True
      #new_index: "<prefix-{now/d}-1>"
      extra_settings:
        index.number_of_shards: 3
        index.number_of_replicas: 1
    filters:
    - filtertype: pattern
      kind: prefix
      value: '^file*'
    - filtertype: age
      source: creation_date
      direction: older
      unit: months
      unit_count: 3

```

and this is the current error:

```auto
> curator --dry-run actions/rollover.yaml
Traceback (most recent call last):
  File "/Users/jroberts235/Library/Python/2.7/bin/curator", line 11, in <module>
    sys.exit(cli())
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/cli.py", line 213, in cli
    run(config, action_file, dry_run)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/cli.py", line 119, in run
    action_dict = validate_actions(action_config)
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/utils.py", line 1410, in validate_actions
    loc
  File "/Users/jroberts235/Library/Python/2.7/lib/python/site-packages/curator/validators/schemacheck.py", line 69,in result
    self.test_what, self.location, self.badvalue, self.error)
curator.exceptions.ConfigurationError: Configuration: structure: Location: Action ID "1": Bad Value: "[{'kind': 'prefix', 'filtertype': 'pattern', 'value': '^file*'}, {'source': 'creation_date', 'direction': 'older', 'filtertype': 'age', 'unit_count': 3, 'unit': 'months'}]", extra keys not allowed @ data['filters']. Check configuration file.

```

I'm ready to thrown in the towel and look for another solution.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [January 16, 2019, 3:22am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/6 "2019-01-16T03:22:14Z")

</div>

Rollover doesn’t use filters because the `name` is an alias. There currently is no way to filter aliases, so Curator can only rollover one alias at a time.

A feature request to allow multiple aliases to be specified is [here](https://github.com/elastic/curator/issues/1266)

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [January 16, 2019, 4:36am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/7 "2019-01-16T04:36:00Z")

</div>

With that said, and me home now, I am taking a closer look:

> [@jroberts235](#):
>
> ```auto
> --- 
> actions:
> 1:
> action: rollover
> options: 
> disable_action: False
> name: filebeat
> conditions:
> #warn_if_no_indices: True
> #new_index: "<prefix-{now/d}-1>"
> extra_settings: 
> index.number_of_shards: 3
> index.number_of_replicas: 1
> 
> ```

This won't work because there are no `conditions`. You need to specify at least one of `max_size`, `max_docs`, or `max_age`.

And this isn't working if it's missing the `action` clause:

```auto
description: >-
  Rollover the index associated with index 'name', which should be in the
  form of prefix-000001 (or similar), or prefix-YYYY.MM.DD-1.
options:
  name: aliasname
  conditions:
    max_age: 1d
    max_docs: 1000000
  extra_settings:
    index.number_of_shards: 3
    index.number_of_replicas: 1
  timeout_override:
  continue_if_exception: False
  disable_action: False

```

A functional merging of the two would be:

```auto
--- 
actions:
  1:
    action: rollover
    options:
      name: filebeat
      conditions:
        max_age: 5d
      extra_settings: 
        index.number_of_shards: 3
        index.number_of_replicas: 1

```

This would rollover at 5 days.

I'm sorry that Curator can't accurately parse YAML structural errors in a pleasing way.

```auto
extra keys not allowed @ data['filters']

```

is YAML linting code for, "There shouldn't be a `filters` key here."

---

<div class="post-metadata">

**Author:** ![jroberts235](https://avatars.discourse-cdn.com/v4/letter/j/cdc98d/32.png) [@jroberts235](https://discuss.elastic.co/u/jroberts235)\
**Post date:** [January 16, 2019, 5:06am UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/8 "2019-01-16T05:06:32Z")

</div>

Thank you for your response.

The "action" that I posted that is in fact missing the "action" statement, is just a cut and paste error from when I posted it. The actual script does indeed include the "action" line, and when run, I do get the error I posted above.

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [January 16, 2019, 2:42pm UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/9 "2019-01-16T14:42:30Z")

</div>

Have you tried the block I shared? If so, what are the error(s)?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 13, 2019, 2:42pm UTC](https://discuss.elastic.co/t/first-attempt-rollover-using-filters-extra-keys-not-allowed-data-filters/163454/10 "2019-02-13T14:42:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.