We have an ELK server that periodically runs out of space. In the spring I worked out a method using curator to delete indices that were more than 30 days old.
But I still don't know much about ELK - Because everyone is working from home nowadays, I can't walk over to the devs and ask. But I still need to solve this problem.
When I run my curator, it frees up the disk space by deleting indices - but many of the indices (and stored queries apparently) are still needed and need to be manually recreated. I figure I could script the creation of the needed files using bash and CLI commands, but that isn't ideal. Is there a way to use curator, or some other tool, to just reduce the index size (clean out data more than 30 days old)?
Here is what I have now:
[root@ELK elasticsearch]# cat /etc/curator/config.yml
client:
- hosts:*
-
- 10.X.X.X*
- port: 9200*
- url_prefix: *
- use_ssl: False*
- certificate:*
- client_cert: /etc/elasticsearch/config/certs/elk/elk.crt*
- client_key: /etc/elasticsearch/config/certs/elk/elk.key*
- ssl_no_validate: False*
- http_auth: *
- timeout: 30*
- master_only: False*
logging:
- loglevel: INFO*
- logfile: /var/log/curator/curator_log*
- logformat: default*
- blacklist: ['elasticsearch', 'urllib3']*
[root@ELK elasticsearch]# cat /etc/curator/action.yml
actions:
- 1:*
- action: delete_indices*
- description: >-*
-
Delete indices older than 30 days (based on creation date).* - options:*
-
ignore_empty_list: True* -
disable_action: False* - filters:*
-
- filtertype: age*
-
source: creation_date* -
direction: older* -
timestring: '%Y.%m.%d'* -
unit: days* -
unit_count: 30*
Any help would be very much appreciated!