Purging when storage is low

[jimyoo]

I want to develop a SW (c++ or script) that checks the available (cluster level) disk space and deletes old indices if the space is under certain amount. For example, if 85% or more of disk has been used, it deletes the oldest index.
I can think of using _cluster/stats to get available space, _cat/indices to get the list of indices, and DETLET to delete old one(s).
Better ideas or/and comments?
(ES version 5.0)

Thanks in advance.

[warkolm (Mark Walkom)]

Elasticsearch Curator does all that and more.

[jimyoo]

Thanks for the information.

I found a command like under. Is this "space 10TB" per cluster (not per index)?
curator.py --host my-elasticsearch -C space -g 10024
I need an option for per cluster.

[warkolm (Mark Walkom)]

What version are you running?

[jimyoo]

I haven't installed Curator yet; just reading articles.

Curator: Tending your time-series indices | Elastic

ES is 5.0.

[warkolm (Mark Walkom)]

You probably want to look at https://www.elastic.co/guide/en/elasticsearch/client/curator/current/filtertype_space.html instead.

[jimyoo]

I ran some tests and it seems to be a per index operation - with 1G disk_space filter, 3 indices with bigger than 1G got removed.

-- purge_on_low_disk.yml --
actions:
1:
action: delete_indices
description: "purging on disk low"
options:
timeout_override:
continue_if_exception: False
disable_action: False
filters:
- filtertype: space
disk_space: 1
reverse: True
use_age: False
source: creation_date
timestring:
field:
stats_result:
exclude: False

$ curator --dry-run purge_on_low_disk.yml --config curator.yml
--- snip ---
2017-01-04 15:22:15,895 DEBUG curator.indexlist __not_actionable:39 Index event-2016.12.12 is not actionable, removing from list.
2017-01-04 15:22:15,895 DEBUG curator.indexlist __excludify:58 Removed from actionable list: event-2016.12.12, summed disk usage is 932.6MB and disk limit is 1.0GB.
2017-01-04 15:22:15,896 DEBUG curator.indexlist __not_actionable:39 Index event-2016.12.11 is not actionable, removing from list.
2017-01-04 15:22:15,896 DEBUG curator.indexlist __excludify:58 Removed from actionable list: event-2016.12.11, summed disk usage is 1018.2MB and disk limit is 1.0GB.
2017-01-04 15:22:15,896 DEBUG curator.indexlist __actionable:35 Index event-2016.12.10 is actionable and remains in the list.
2017-01-04 15:22:15,896 DEBUG curator.indexlist __excludify:58 Remains in actionable list: event-2016.12.10, summed disk usage is 1.1GB and disk limit is 1.0GB.
2017-01-04 15:22:15,896 DEBUG curator.indexlist __actionable:35 Index event-2016.12.09 is actionable and remains in the list.
2017-01-04 15:22:15,897 DEBUG curator.indexlist __excludify:58 Remains in actionable list: event-2016.12.09, summed disk usage is 1.2GB and disk limit is 1.0GB.
2017-01-04 15:22:15,897 DEBUG curator.indexlist __actionable:35 Index event-2016.12.08 is actionable and remains in the list.
2017-01-04 15:22:15,897 DEBUG curator.indexlist __excludify:58 Remains in actionable list: event-2016.12.08, summed disk usage is 1.2GB and disk limit is 1.0GB.
2017-01-04 15:22:15,897 DEBUG curator.utils iterate_filters:847 Post-instance: ['event-2016.12.08', 'event-2016.12.10', 'event-2016.12.09']
2017-01-04 15:22:15,898 DEBUG curator.actions.delete_indices init:421 master_timeout value: 30s
2017-01-04 15:22:15,898 INFO curator.utils show_dry_run:636 DRY-RUN MODE. No changes will be made.
2017-01-04 15:22:15,898 INFO curator.utils show_dry_run:639 (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2017-01-04 15:22:15,898 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: event-2016.12.08 with arguments: {}
2017-01-04 15:22:15,898 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: event-2016.12.09 with arguments: {}
2017-01-04 15:22:15,898 INFO curator.utils show_dry_run:646 DRY-RUN: delete_indices: event-2016.12.10 with arguments: {}