Log Retention

[motts]

Does Elasticsearch have a default log retention period?

If so, how do I change that.

If not, what is the best way to go about purging old logs?

Many thanks in advance.

[A_B]

As far as I know, the preferred way to manage indices and retention is to use curator
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about-features.html

[motts]

Thanks for the info. I am trying to set up curator based off the example, but when I try the --dry-run, I get the following error:

Unable to parse YAML file. Error: mapping values are not allowed here
  in "<unicode string>", line 20, column 9:
        kind: prefix
            ^

I am not sure what's causing the error. I have checked a couple other configs and they appear to be the same. Here is my config file.

actions:
1:
action: delete_indices
description: >-
 Delete indices older than 2 days (based on index name), for logstash-prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.
options:
 ignore_empty_list: True
 timeout_override:
 continue_if_exception: False
 disable_action: False
filters:
 - filtertype: pattern
    kind: prefix
    value: logstash-
    exclude:
 - filtertype: age
    source: name
    direction: older
    timestring: '%Y.%m.%d'
    unit: days
    unit_count: 2
    exclude:

[theuntergeek (Aaron Mildenstein)]

You inappropriately indented kind and everything beneath it. YAML is very particular about indentation. Your file should look like:

---
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 2 days (based on index name), for logstash-prefixed indices. Ignore the error 
      if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. 
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-
        exclude:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 2
        exclude:

[motts]

Ok. I tried this:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# # not a Python "NoneType"
# #
# # Also remember that all examples have 'disable_action' set to True.  If you
# # want to use this action as a template, be sure to set this to False after
# # copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 45 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: True
    filters:
      - filtertype: pattern
        kind: prefix
        value: logstash-
        exclude:
      - filtertype: age
        source: name
        direction: older
        timestring: '%Y.%m.%d'
        unit: days
        unit_count: 45
        exclude:

but now I get this error:
[root@localhost ~]# curator --dry-run /root/.curator/curator.yml
Schema error: extra keys not allowed @ data['actions']
Configuration: Client Configuration: Location: full configuration dictionary: Bad Value: "{1: {'action': 'delete_indices', 'description': 'Delete indices older than 45 days (based on index name), for logstash- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly.', 'options': {'ignore_empty_list': True, 'timeout_override': None, 'continue_if_exception': False, 'disable_action': True}, 'filters': [{'filtertype': 'pattern', 'kind': 'prefix', 'value': 'logstash-', 'exclude': None}, {'filtertype': 'age', 'source': 'name', 'direction': 'older', 'timestring': '%Y.%m.%d', 'unit': 'days', 'unit_count': 45, 'exclude': None}]}}", extra keys not allowed @ data['actions']. Check configuration file.

[motts]

I've also tried following the model at https://www.elastic.co/guide/en/elasticsearch/client/curator/4.2/ex_delete_indices.html with the "-" under the f in"filters:" and moved everything to line up, but I still get the same error message "...extra keys not allowed @ data ['actions']. Check configuration file."

[theuntergeek (Aaron Mildenstein)]

Ah:

@motts

Schema error: extra keys not allowed @ data['actions']
Configuration: Client Configuration:

That error is telling you that you specified the same file as both an action file and a client configuration file.

Your client definition should be at /root/.curator/curator.yml, and your action file (the part you shared above) should be, well, anywhere else, and not named curator.yml to make it easier to distinguish.

Here is a client configuration example:

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile:
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

This is what tells Curator where your cluster is, and how it should connect, log, etc. It is different from the action file.

[motts]

Awesome, worked like a charm. Thanks a million.