Error when running curator

Elastic Stack Elasticsearch
1

Hello everyone,

I'm an absolute beginner who encounters many problems and haven't find a solution yet.

I've been running Elasticsearch and kibana for some months on a debian 10 VM. Elasticsearch receives logs from 2 winlogbeat agents and I can see everything I want with Kibana (I am not using logstash). That was great, until the space disk of my VM got full (duh !). I obviously couldn't see anything anymore on kibana, and after some testings I was not able to restart properly the kibana service either. I assume that's also because the disk was full.

That is the reason I'm using curator. Or at least I'm trying. I would like to delete everything older than 30 days and despite having many examples on internet I could not make it work.

.curator/curator.yml

---
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  username:
  password:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

action.yml

---
actions:
  1:
    action: delete_indices
    description: Delete indices with %Y.%m.%d in the name where that date is older than 30 days
    options:
      ignore_empty_list: True
    filters:
      - filtertype: age
        source: name
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 30

Here is the error I get when I run curator action.yml

2021-10-25 13:41:02,650 ERROR Schema error: required key not provided @ data['direction']
2021-10-25 13:41:02,650 ERROR Schema error: Configuration: filter: Location: Action ID "1", action "delete_indices", filter #0: {'filtertype': 'age', 'source': 'name', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 30}: Bad Value: "(could not determine)", required key not provided @ data['direction']. Check configuration file.

I know other persons had similar problem but no solution I found online worked.

I suspect one of my problem is the "source" parameter. I guess I'm supposed to replace it with something but I don't know what.
Does anyone have an idea ? Did I provide enough information ?

Thanks in advance

2
  1. Having only an age filter with no pattern filter is going to be destructive once your .kibana and .security (and other) indices are older than 30 days. I cannot overemphasize how bad your experience will be with only this filter in place. In simpler terms, this filter will identify ALL indices (presumably) older than 30 days and perform the indicated action (delete_indices) on them, meaning that your .kibana and any other index older than 30 days. I'm 100% certain this is not what you want, so I highly recommend using a pattern filter before the age filter to prevent this from happening to indices you don't want to have deleted.
  2. You have omitted the direction in the age filter. Without this, Curator doesn't know whether you're looking for indices older or younger than the reference point of 30 days. This is what the error message is indicating when it says "required key not provided @data['direction']"
3

Thank you very much, the "direction" missing was indeed the problem. I added a pattern too as you recommanded and everything worked well.

Thanks again !

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.