Monthly backup and restore along with snapshot

Elastic Stack Elasticsearch
1

Hi,

We are pretty new to Elasticseach, so our question may look like something trivial, But really need your advise.

We have Elasticsearch hosted on elastic cloud. We are using this to save our logs.

At any given time, we need to have access to logs for past 30 days in queryable format. And whatever logs are older should get removed automatically to save storage space.
Also we need to have regular snapshot on say hourly/ x hourly basis so in-case of any incident, we can restore those logs (with margin of few lost logs).
Also we may have some situations where we may require logs back in elasticsearch which are say 6 months older. So in this case we would like to restore those logs (and existing logs of past 15 days also remain).

In Summary

  1. Hourly backup/snapshot of logs
  2. Retention of logs say past 15 Days
  3. Any logs older than 15 days gets removed from Elasticsearch
  4. A cron job to take backup of logs every 15 days
  5. Ability to restore from any snapshot (incase of incidents for within 15 days)
  6. Ability to restore logs (complete bunch of 15 days) for any past month keeping existing logs of past 15 days intact.

We needd to save all snapshots and backups in S3 and restore from there.

Thanks for your time reading through this.

2

Hey,

First regarding the hourly/snapshotting. This is the default, see Work with snapshots | Elasticsearch Service Documentation | Elastic

By default, Elasticsearch Service takes a snapshot of all the indices in your Elasticsearch cluster every 30 minutes. You can set a different snapshot interval, if needed for your environment. The snapshot count can keep anywhere between a minimum of 12 or up to a maximum of the 100 most recent, successful snapshots.

  1. You may want to use index lifecycle management for that

5/6. This is all about elasticsearchs snapshot & restore functionality

hope this helps as a start!

3

Hi,

Thanks for your response, really appreciate it.
One question though, lets say we take advantage of warm and cold indices.

  1. In Index Lifecycle, we set duration for Warm and Cold indexes. So after configured timeperiod, indexes move to cold from warm.
  2. Now before deleting it, can we take a complete backup of that cold index so when required we can restore it completely? As what I understand, Snapshots are incremental and might not be helpfull if we want to restore an index which was deleted say 6 months back.

I am bit confused around it, please advise.

Thanks again for your time and response.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.