We are pretty new to Elasticseach, so our question may look like something trivial, But really need your advise.
We have Elasticsearch hosted on elastic cloud. We are using this to save our logs.
At any given time, we need to have access to logs for past 30 days in queryable format. And whatever logs are older should get removed automatically to save storage space.
Also we need to have regular snapshot on say hourly/ x hourly basis so in-case of any incident, we can restore those logs (with margin of few lost logs).
Also we may have some situations where we may require logs back in elasticsearch which are say 6 months older. So in this case we would like to restore those logs (and existing logs of past 15 days also remain).
- Hourly backup/snapshot of logs
- Retention of logs say past 15 Days
- Any logs older than 15 days gets removed from Elasticsearch
- A cron job to take backup of logs every 15 days
- Ability to restore from any snapshot (incase of incidents for within 15 days)
- Ability to restore logs (complete bunch of 15 days) for any past month keeping existing logs of past 15 days intact.
We needd to save all snapshots and backups in S3 and restore from there.
Thanks for your time reading through this.