Periodically over the last year or two, I have taken snapshots of different indices. Some of these I have recorded in a spreadsheet, some I have not. Now, I would like to delete all snapshots older then 90 days that do not include the prefix 'citydirectory', but without having to go through the spreadsheet and deleting them one by one. Therefore by these requirements: a snapshot that's 6 months old called 'citydirectory_v1' would not be deleted, but another snapshot 'example2' that's also 6 months old would be deleted. What is the best way to do something like this? I was thinking of writing a python script that returns all snapshots on s3, but this may not include the dates. I also think there is an intuitive was to do this in Elasticsearch, or as part of the ElasticStack, although I can't remember what it is. If you think of a better way to do this, please let me know. Thank you.
you could try Elastic Curators delete Snapshot funtionality
Hey @Shaoranlaos, I'm taking a look at curator now. I will post my results here later. Thanks for the tip!
Hi @Shaoranlaos I have setup curator and have my script -- it's show below:
Curator.yml
client:
hosts:
- 'a data node'
port: 9200
url_prefix:
use_ssl: False
certificate:
client_cert:
client_key:
ssl_no_validate: False
http_auth:
timeout: 30
master_only: Falselogging:
loglevel: INFO
logfile: D:\CuratorLogs\logs.txt
logformat: default
blacklist: ['elasticsearch', 'urllib3']
delete_old_snapshots.yml
actions:
1:
action: delete_snapshots
description: >-
Delete snapshots from the selected repository older than 100 days
(based on creation_date), for everything but 'citydirectory-' prefixed snapshots.
options:
repository: 'dcs-elastic-snapshot'
disable_action: False
filters:
- filtertype: pattern
kind: prefix
value: citydirectory-
exclude: True
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 100
For some reason I keep getting the exception 'Failed to complete action: delete_snapshots. <class 'curator.exceptions.FailedExecution'>: Unable to verify existence of repository dcs-elastic-snapshot
'. Any idea why it's not finding the correct s3 repository?
I figured it out. In repository - put the name you registered the s3 bucket as, not as what it appears in s3 if they differ. So for me I registered my s3 bucket under 'dcselasticsnapshot' rather then what is appears in s3 'dcs-elastic-snapshot'.
fyi this solves my question, so marking this as the correct answer.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.