Dynamic index expiration dates

Elastic Stack Elasticsearch
1

We're using the following curator script to expire data older than 7 days.

actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 7 days (based on index name), for logstash-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
      exclude:
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 7
      exclude:

We have data from several different users, we would like some users to have a different index retention than others. What would we the best way to achieve this?

Right now the names of our indices start with logstash-Y.m.d

One way I thought it would be possible is to have several indices depending on the retention, for example for 14 day retention logstash14d-%Y.%m.%d for 30 days logstash30d-%Y.%m.%d

The downside to this is that we would have to check before ingesting every log event what the users retention is.

1 Like
2

Retention at the index level can only be achieved with different named indices. You correctly understand this. And, yes, it also means you have to figure out how to route that data in your Logstash configuration.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.