Curator field_stats doesn't work

Elastic Stack Elasticsearch
1

I updated curator from 5.5.2 to 5.5.4

Before that:

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field": "@timestamp","stats_result":"max_value"}]'

worked well

Now it throws:

ERROR     curator.validators.SchemaCheck                 result:65   Schema error: extra keys not allowed @ data['field']
ERROR     curator.validators.SchemaCheck                 result:65   Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL  curator.cli_singletons.cli_action.show_indices          check_filters:126  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file

config:

client:
  hosts: 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /tmp/t.log
  logformat: default

I read:

In Curator 5.3 and older, source field_stats uses the Field Stats API to calculate either the min_value or the max_value of the field as the stats_result , and then use that value for age comparisons. In 5.4 and above, even though it is still called field_stats , it uses an aggregation to calculate the same values, as the field_stats API is no longer used in Elasticsearch 6.x and up.

field must be of type date in Elasticsearch.

But what does that mean for me to get the same functionality as before the update?

2

The type date part shouldn't matter, as that's not what's causing your error.

It is complaining about the key "field" in your age filter. I wonder if it's because there's a space after "field": "@timestamp".

Does it do the same with a regular YAML action file?

3

Interesting.

curator  --config config --dry-run action

action

actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: field_stats
      direction: older
      field: '@timestamp'
      stats_result: min_value
      unit: days
      unit_count: 200
4 
DEBUG                curator.cli                    run:108  Client and logging options validated.
DEBUG                curator.cli                    run:112  default_timeout = 30
DEBUG                curator.cli                    run:116  action_file: action
DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4268>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4400>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4598>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4730>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'ignore_empty_list': True, 'disable_action': True}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function Filters.<locals>.f at 0x7f72beb34400>
DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(<class 'str'>, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4bf8>, msg=None), msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}
DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}
5 
DEBUG     curator.defaults.filtertypes                    age:55   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'field': Any(<class 'str'>, msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}
DEBUG     curator.validators.filters                      f:48   Filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}
DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True, 'continue_if_exception': False, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}]}}
DEBUG                curator.cli                    run:126  action_disabled = True
DEBUG                curator.cli                    run:130  continue_if_exception = False
DEBUG                curator.cli                    run:132  timeout_override = None
DEBUG                curator.cli                    run:134  ignore_empty_list = True
DEBUG                curator.cli                    run:136  allow_ilm_indices = False
INFO                 curator.cli                    run:142  Action ID: 1: "delete_indices" not performed because "disable_action" is set to True
INFO                 curator.cli                    run:197  Job completed.
6

Does it behave differently with the space removed from "field": "@timestamp"?

e.g.

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field":"@timestamp","stats_result":"max_value"}]'
7 
ERROR     Schema error: extra keys not allowed @ data['field']
ERROR     Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file.
8

@timestamp is the default value of field, so technically it may not be necessary. Test this with a --dry-run, of course, but it should work. And in the meanwhile, I will be trying to find why this is not applying the schema validation properly.

9

Thanks for your support. The next problem would be stats_result :smiley:

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"stats_result":"max_value"}]' 
ERROR     Schema error: extra keys not allowed @ data['stats_result']
ERROR     Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.
CRITICAL  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.. Check configuration file.
10

Can you at least reproduce the problem locally?

11

I'll take a look at it. Please add this as an issue at https://github.com/elastic/curator/issues, as this sounds like a bug, and I'd like it to be tracked there.

Sorry for the delays. At Elastic, I'm no longer a developer as my day job. I switched to the Professional Services team as a Consultant over a year ago, and that means my coding time is sometimes reduced do to the demands of my new position. I just don't always have time to work on Curator like I used to.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.