# Curator field\_stats doesn't work

**URL:** <https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619>\
**Category:** Elasticsearch\
**Created:** [September 6, 2018, 5:54pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619 "2018-09-06T17:54:53Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 5:54pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/1 "2018-09-06T17:54:53Z")

</div>

I updated curator from 5.5.2 to 5.5.4

Before that:

```auto
curator_cli --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field": "@timestamp","stats_result":"max_value"}]'

```

worked well

Now it throws:

```auto
ERROR curator.validators.SchemaCheck result:65 Schema error: extra keys not allowed @ data['field']
ERROR curator.validators.SchemaCheck result:65 Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL curator.cli_singletons.cli_action.show_indices check_filters:126 Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file

```

config:

```auto
client:
  hosts: 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: DEBUG
  logfile: /tmp/t.log
  logformat: default

```

I read:

> In Curator 5.3 and older, source `field_stats` uses the [Field Stats API](http://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-field-stats.html) to calculate either the `min_value` or the `max_value` of the [`field`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/fe_field.html) as the [`stats_result`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/fe_stats_result.html), and then use that value for age comparisons. In 5.4 and above, even though it is still called `field_stats` , it uses an aggregation to calculate the same values, as the `field_stats` API is no longer used in Elasticsearch 6.x and up.
> 
> [`field`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/fe_field.html) must be of type `date` in Elasticsearch.

But what does that mean for me to get the same functionality as before the update?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 6, 2018, 6:10pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/2 "2018-09-06T18:10:52Z")

</div>

The type `date` part shouldn't matter, as that's not what's causing your error.

It is complaining about the key "field" in your age filter. I wonder if it's because there's a space after `"field": "@timestamp"`.

Does it do the same with a regular YAML action file?

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 7:19pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/3 "2018-09-06T19:19:54Z")

</div>

Interesting.

```auto
curator --config config --dry-run action 

```

action

```auto
actions:
  1:
    action: delete_indices
    options:
      ignore_empty_list: True
      disable_action: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: field_stats
      direction: older
      field: '@timestamp'
      stats_result: min_value
      unit: days
      unit_count: 200

```

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 7:20pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/4 "2018-09-06T19:20:36Z")

</div>

```auto
DEBUG curator.cli run:108 Client and logging options validated.
DEBUG curator.cli run:112 default_timeout = 30
DEBUG curator.cli run:116 action_file: action
DEBUG curator.cli run:118 action_config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'actions': <class 'dict'>}
DEBUG curator.validators.SchemaCheck __init__ :27 "Actions File" config: {'actions': {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}}}
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']")}
DEBUG curator.validators.SchemaCheck __init__ :27 "action type" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'action': Any(In(['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']), msg="action must be one of ['alias', 'allocation', 'close', 'cluster_routing', 'create_index', 'delete_indices', 'delete_snapshots', 'forcemerge', 'index_settings', 'open', 'reindex', 'replicas', 'restore', 'rollover', 'shrink', 'snapshot']"), 'description': Any(<class 'str'>, <class 'str'>, msg=None), 'options': <class 'dict'>, 'filters': <class 'list'>}
DEBUG curator.validators.SchemaCheck __init__ :27 "structure" config: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]}
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'allow_ilm_indices': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4268>, msg=None), msg=None), 'continue_if_exception': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4400>, msg=None), msg=None), 'disable_action': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4598>, msg=None), msg=None), 'ignore_empty_list': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4730>, msg=None), msg=None), 'timeout_override': Any(Coerce(int, msg=None), None, msg=None)}
DEBUG curator.validators.SchemaCheck __init__ :27 "options" config: {'ignore_empty_list': True, 'disable_action': True}
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: <function Filters.<locals>.f at 0x7f72beb34400>
DEBUG curator.validators.SchemaCheck __init__ :27 "filters" config: [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}]
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'kind': Any('prefix', 'suffix', 'timestring', 'regex', msg=None), 'value': Any(<class 'str'>, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beac4bf8>, msg=None), msg=None)}
DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-'}
DEBUG curator.validators.filters f:48 Filter #0: {'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}

```

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 7:20pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/5 "2018-09-06T19:20:45Z")

</div>

```auto
DEBUG curator.defaults.filtertypes age:55 AGE FILTER = [{'direction': Any('older', 'younger', msg=None)}, {'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None)}, {'unit_count': Coerce(int, msg=None)}, {'unit_count_pattern': Any(<class 'str'>, msg=None)}, {'epoch': Any(Coerce(int, msg=None), None, msg=None)}, {'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None)}, {'source': Any('name', 'creation_date', 'field_stats', msg=None)}, {'stats_result': Any('min_value', 'max_value', msg=None)}, {'field': Any(<class 'str'>, msg=None)}, {'timestring': Any(None, <class 'str'>, msg=None)}]
DEBUG curator.validators.SchemaCheck __init__ :26 Schema: {'filtertype': Any(In(['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']), msg="filtertype must be one of ['age', 'alias', 'allocated', 'closed', 'count', 'forcemerged', 'ilm', 'kibana', 'none', 'opened', 'pattern', 'period', 'space', 'state']"), 'direction': Any('older', 'younger', msg=None), 'unit': Any('seconds', 'minutes', 'hours', 'days', 'weeks', 'months', 'years', msg=None), 'unit_count': Coerce(int, msg=None), 'unit_count_pattern': Any(<class 'str'>, msg=None), 'epoch': Any(Coerce(int, msg=None), None, msg=None), 'exclude': Any(<class 'bool'>, All(Any(<class 'str'>, msg=None), <function Boolean at 0x7f72beaec7b8>, msg=None), msg=None), 'source': Any('name', 'creation_date', 'field_stats', msg=None), 'stats_result': Any('min_value', 'max_value', msg=None), 'field': Any(<class 'str'>, msg=None), 'timestring': Any(None, <class 'str'>, msg=None)}
DEBUG curator.validators.SchemaCheck __init__ :27 "filter" config: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200}
DEBUG curator.validators.filters f:48 Filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}
DEBUG curator.cli run:121 Full list of actions: {1: {'action': 'delete_indices', 'options': {'ignore_empty_list': True, 'disable_action': True, 'continue_if_exception': False, 'allow_ilm_indices': False, 'timeout_override': None}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'filebeat-', 'exclude': False}, {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'field': '@timestamp', 'stats_result': 'min_value', 'unit': 'days', 'unit_count': 200, 'exclude': False, 'timestring': None, 'epoch': None}]}}
DEBUG curator.cli run:126 action_disabled = True
DEBUG curator.cli run:130 continue_if_exception = False
DEBUG curator.cli run:132 timeout_override = None
DEBUG curator.cli run:134 ignore_empty_list = True
DEBUG curator.cli run:136 allow_ilm_indices = False
INFO curator.cli run:142 Action ID: 1: "delete_indices" not performed because "disable_action" is set to True
INFO curator.cli run:197 Job completed.

```

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 6, 2018, 8:30pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/6 "2018-09-06T20:30:29Z")

</div>

Does it behave differently with the space removed from `"field": "@timestamp"`?

e.g.

```auto
curator_cli --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"field":"@timestamp","stats_result":"max_value"}]'

```

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 8:49pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/7 "2018-09-06T20:49:51Z")

</div>

```auto
ERROR Schema error: extra keys not allowed @ data['field']
ERROR Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.
CRITICAL Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'field': '@timestamp', 'stats_result': 'max_value'}: Bad Value: "@timestamp", extra keys not allowed @ data['field']. Check configuration file.. Check configuration file.

```

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 6, 2018, 9:31pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/8 "2018-09-06T21:31:04Z")

</div>

`@timestamp` is the default value of `field`, so technically it may not be necessary. Test this with a `--dry-run`, of course, but it should work. And in the meanwhile, I will be trying to find why this is not applying the schema validation properly.

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 6, 2018, 9:49pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/9 "2018-09-06T21:49:37Z")

</div>

Thanks for your support. The next problem would be stats\_result 😃

```auto
curator_cli --config config show_indices --filter_list '[{"filtertype":"pattern","kind":"prefix","value":"filebeat-"}, {"filtertype":"age","source":"field_stats","direction":"older","unit":"days","unit_count":200,"stats_result":"max_value"}]' 
ERROR Schema error: extra keys not allowed @ data['stats_result']
ERROR Schema error: Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.
CRITICAL Unable to parse filters: Configuration: filters: Location: show_indices singleton action "filters": Bad Value: "None", Configuration: filter: Location: singleton, filter #1: {'filtertype': 'age', 'source': 'field_stats', 'direction': 'older', 'unit': 'days', 'unit_count': 200, 'stats_result': 'max_value'}: Bad Value: "max_value", extra keys not allowed @ data['stats_result']. Check configuration file.. Check configuration file.

```

---

<div class="post-metadata">

**Author:** ![aubreybox](https://avatars.discourse-cdn.com/v4/letter/a/ecccb3/32.png) [@aubreybox](https://discuss.elastic.co/u/aubreybox)\
**Post date:** [September 9, 2018, 11:29am UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/10 "2018-09-09T11:29:46Z")

</div>

Can you at least reproduce the problem locally?

---

<div class="post-metadata">

**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\
**Post date:** [September 10, 2018, 1:41pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/11 "2018-09-10T13:41:38Z")

</div>

I'll take a look at it. Please add this as an issue at [https://github.com/elastic/curator/issues](https://github.com/elastic/curator/issues), as this sounds like a bug, and I'd like it to be tracked there.

Sorry for the delays. At Elastic, I'm no longer a developer as my day job. I switched to the Professional Services team as a Consultant over a year ago, and that means my coding time is sometimes reduced do to the demands of my new position. I just don't always have time to work on Curator like I used to.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 8, 2018, 1:41pm UTC](https://discuss.elastic.co/t/curator-field-stats-doesnt-work/147619/12 "2018-10-08T13:41:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.