Curator Older Than configuration does not work

Elastic Stack Elasticsearch
1

Hello,

I' m trying to delete logs older than 10 days within an indice, based on the timestamp of my logs indice, but it does not seem to work as curator seems to delete logs older than 5 days instead,

The name of my indice is as follow : indice-2018.04

Structure of this indice contains following fields :

@timestamp April 14th 2018, 19:07:01.000

And in curator my configuration is :

  2:
action: delete_indices
description: >-
  blabla
options:
  ignore_empty_list: True
  timeout_override:
  continue_if_exception: False
  disable_action: False
filters:
- filtertype: pattern
  kind: prefix
  value: indice-
  exclude:
- filtertype: age
  source: field_stats
  field: '@timestamp'
  stats_result: min_value
  direction: older
  unit: days
  unit_count: 10
  exclude:

Could you me seing what is wrong there?

2

If you run with --dry-run and loglevel: DEBUG you'll see everything, including the calculated time values. Those log lines will be important to help figure out what's happening.

3

Curator manages retention by deleting complete indices, not individual documents within an index. Based on your index name it looks like you are using monthly indices, so I do not see how you will be able to delete just data that is older than 10 days with this.

4

Thx, indeed I was asking me the question,

I thought thas as curator can be configurated to delete "objects" based on timestamp, it would be on logs timestamp, therefore it could only delete logs older than ...

So you are saying to me that curator can only delete complete indices.

is there another solution to delete only specific logs of a dedicated indice? or do I have no other choice than having daily indices?

5

You should not be deleting from an index if you are using time-series data. Retention should be managed by dropping entire indices, rather than trying to delete from an index. The "drop" is a single operation, while the delete from is millions of atomic operations, which is extremely taxing for the I/O operations of each node that has a shard for that index.

If you need to be able to search across multiple indices to cover all of the data you have, use aliases, or index patterns to search across all indices.

6

Ok thanks for your answer,

I already have an index pattern matching the prefix "indice-",

So maybe I could start adding indexes containing also day date, and I would then be able to delete only indexes older than 10 days,

It shouldnt break my index pattern if I do it that way dont you think?

7

No. It wouldn't.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.