Deleting indices older than 90 days

Elastic Stack Elasticsearch
1

Hi i am trying to delete indices older than 90 days.
My elastic search index pattern is indexname-%Y.%m in this case it is applicationlogs-2017-12
Below is my action.yml
actions:
1:
action: delete_indices
description: >-
Delete applicationlogs indices older than given days
options:
ignore_empty_list: True
filters:
- filtertype: pattern
kind: prefix
value: applicationlogs-
- filtertype: age
source: name
direction: older
timestring: '%Y.%m'
unit: days
unit_count: 90

It is deleting the indices not by days month wise. If i am having indices as
applicationlogs-2017-11
applicationlogs-2017-12
applicationlogs-2018-01
applicationlogs-2018-02

action.yml is deleting november month full indices [ From 1st november to 30 november].

This behavior Is because i am having only month in the index? Please need some pointers.
I am using curator, version 5.4.1

2

maybe you can try to change the unit: days to unit: months, and change unit_count: 90 to unit_count: 3.

3

curator works only on index level not on document level

so curator can not delete documents within an index
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/about-features.html

4

can use

kind: regex
source: creation_date

5

Thanks for the reply. Still it is deleting full month data.
Example if 90 days are getting over on January 10 from today ideally it should delete from 10 and old data but here it is deleting full January month data too.

6

Thanks for the reply. But still it is deleting full month data. May be it is because we are creating index based on month.

7

as this. use --dry-run test which indices delete .
kind: prefix
this can do many.. you can try .

8

In dry run giving following results when i do --dry-dun
08:53:31,412 INFO Preparing Action ID: 1, "delete_indices"
08:53:31,419 INFO Trying Action ID: 1, "delete_indices": Delete inforlogs indices older than given days
08:53:31,586 INFO DRY-RUN MODE. No changes will be made.
08:53:31,586 INFO (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
08:53:31,587 INFO DRY-RUN: delete_indices: applicationlogs-2018.01 with arguments: {}
08:53:31,587 INFO Action ID: 1, "delete_indices" completed.
08:53:31,587 INFO Job completed.

As per logs it is deleting January month full data.

9

this delete only applicationlogs-2018.01
is not ok??
you can create crontab running moths. not ok?

10

Yes, i am also cron job for deletion.
If today my cron runs and it checks for 90 days it will be middle of the month for example if 90 days is January 5th, the it should not delete January 6th logs but in this case it is deleting.

11

it about indices not docs as i know..

12

Oh ok. Then can i find 6th January document in the elastic search. If yes how can i see? I am new to elasticsearch.

13

if as days . you can change the indices to applicationlogs-2018.01.xx

14

use kibana see the data is better.

15

Yes we have Kibana as log viewer when i delete January index not able to see in the Kibana ui for January .

16

I think the problem you are encountering is that is that Curator is only looking to see if the index date, as indicated by the time string in the index name, is older than 90 days from the moment of execution.

Here's a little python code to show you what's happening under the hood in Curator:

$ python3
Python 3.6.3 (default, Oct  9 2017, 12:04:21)
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.37)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from datetime import datetime, timedelta
>>> datetime.utcnow()
datetime.datetime(2018, 3, 7, 13, 44, 53, 139826)
>>> datetime.utcnow()-timedelta(days=90)
datetime.datetime(2017, 12, 7, 13, 44, 58, 401248)
>>> (datetime.utcnow()-timedelta(days=90)).strftime('%Y.%m.%d')
'2017.12.07'

So, what you see here is that similar to Curator, I'm using utc time, and that 90 days go from right this moment in UTC time is December 7, 2017. Since the rest of the month of December is not yet older than 90 days, Curator will not delete the index applicationlogs-2017-12. With the configuration you provided, Curator will not delete an index until every day of the index (you specified monthly indices) is older than 90 days.

Do you still want Curator to delete applicationlogs-2017-12 even though the dates of December 1-7 are all that are older than 90 days? We can discuss alternate configurations which can achieve something like this if you are interested.

17

Thank you for the reply, I dont want to delete full december month data if 90 days over on december 7th it should delete from 7th and it's previous data, but in my case it is deleting full december month data. When i saw from kibana i am not able to see december month data. If any options to achieve day wise delete let me know i will try.

18

As has been previously mentioned, you cannot use Curator to delete data from an index. It can only delete entire indices.

You should never delete data from an index unnecessarily, especially when it is time series data. Deleting entire indices is the appropriate method for data clean up. If you need to delete older data, you should consider daily or weekly indices, and then your retention will be more exact than monthly.

19

Thank you for the clarification. Now understood fully. We will make changes as per your suggestion.

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.