Curator field_stats doesn't work

aubreybox · September 6, 2018, 5:54pm

I updated curator from 5.5.2 to 5.5.4

Before that:

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field": "@timestamp","stats_result":"max_value"}]'

worked well

Now it throws:

ERROR     curator.validators.SchemaCheck                 result:65   Schema error: extra keys not allowed @ data['field']
ERROR     curator.validators.SchemaCheck                 result:65   Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL  curator.cli_singletons.cli_action.show_indices          check_filters:126  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file

config:

client:
  hosts: 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /tmp/t.log
  logformat: default

I read:

In Curator 5.3 and older, source field_stats uses the Field Stats API to calculate either the min_value or the max_value of the field as the stats_result , and then use that value for age comparisons. In 5.4 and above, even though it is still called field_stats , it uses an aggregation to calculate the same values, as the field_stats API is no longer used in Elasticsearch 6.x and up.

field must be of type date in Elasticsearch.

But what does that mean for me to get the same functionality as before the update?

theuntergeek · September 6, 2018, 6:10pm

The type date part shouldn't matter, as that's not what's causing your error.

It is complaining about the key "field" in your age filter. I wonder if it's because there's a space after "field": "@timestamp".

Does it do the same with a regular YAML action file?

aubreybox · September 6, 2018, 7:19pm

Interesting.

curator  --config config --dry-run action

action

actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: field_stats
      direction: older
      field: '@timestamp'
      stats_result: min_value
      unit: days
      unit_count: 200

aubreybox · September 6, 2018, 7:20pm

DEBUG                curator.cli                    run:108  Client and logging options validated.
DEBUG                curator.cli                    run:112  default_timeout = 30
DEBUG                curator.cli                    run:116  action_file: action
DEBUG                curator.cli                    run:118  action_config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'actions': <class 'dict'>}
DEBUG     curator.validators.SchemaCheck               __init__:27   "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
DEBUG     curator.validators.SchemaCheck               __init__:27   "action type" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
DEBUG     curator.validators.SchemaCheck               __init__:27   "structure" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4268>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4400>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4598>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4730>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "options" config: {'ignore_empty_list': True, 'disable_action': True}
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: <function Filters.<locals>.f at 0x7f72beb34400>
DEBUG     curator.validators.SchemaCheck               __init__:27   "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(<class 'str'>, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4bf8>, msg=None), msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}
DEBUG     curator.validators.filters                      f:48   Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}

aubreybox · September 6, 2018, 7:20pm

DEBUG     curator.defaults.filtertypes                    age:55   AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'field': Any(<class 'str'>, msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
DEBUG     curator.validators.SchemaCheck               __init__:26   Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
DEBUG     curator.validators.SchemaCheck               __init__:27   "filter" config: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}
DEBUG     curator.validators.filters                      f:48   Filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}
DEBUG                curator.cli                    run:121  Full list of actions: {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True, 'continue_if_exception': False, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}]}}
DEBUG                curator.cli                    run:126  action_disabled = True
DEBUG                curator.cli                    run:130  continue_if_exception = False
DEBUG                curator.cli                    run:132  timeout_override = None
DEBUG                curator.cli                    run:134  ignore_empty_list = True
DEBUG                curator.cli                    run:136  allow_ilm_indices = False
INFO                 curator.cli                    run:142  Action ID: 1: "delete_indices" not performed because "disable_action" is set to True
INFO                 curator.cli                    run:197  Job completed.

theuntergeek · September 6, 2018, 8:30pm

Does it behave differently with the space removed from "field": "@timestamp"?

e.g.

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field":"@timestamp","stats_result":"max_value"}]'

aubreybox · September 6, 2018, 8:49pm

ERROR     Schema error: extra keys not allowed @ data['field']
ERROR     Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file.

theuntergeek · September 6, 2018, 9:31pm

@timestamp is the default value of field, so technically it may not be necessary. Test this with a --dry-run, of course, but it should work. And in the meanwhile, I will be trying to find why this is not applying the schema validation properly.

aubreybox · September 6, 2018, 9:49pm

Thanks for your support. The next problem would be stats_result

curator_cli  --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"stats_result":"max_value"}]' 
ERROR     Schema error: extra keys not allowed @ data['stats_result']
ERROR     Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.
CRITICAL  Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.. Check configuration file.

aubreybox · September 9, 2018, 11:29am

Can you at least reproduce the problem locally?

theuntergeek · September 10, 2018, 1:41pm

I'll take a look at it. Please add this as an issue at https://github.com/elastic/curator/issues, as this sounds like a bug, and I'd like it to be tracked there.

Sorry for the delays. At Elastic, I'm no longer a developer as my day job. I switched to the Professional Services team as a Consultant over a year ago, and that means my coding time is sometimes reduced do to the demands of my new position. I just don't always have time to work on Curator like I used to.

system · October 8, 2018, 1:41pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.