Snapshot Policy Based on Data Phase

Elastic Stack Elasticsearch
1

Hi All,

I was wondering if it's possible to create snapshot policies that include/exclude indices based on their phase? I wasn't able to find any documentation on this and didn't see any posts about it.

The use case I am interested in is the following:

Have two (2) S3 repositories:

  • S3_Repo_A is on-prem and has a 25Gbps connection to the cluster
  • S3_Repo_B is in the cloud and has a 1Gbps connection to the cluster

I would like to back up all indexes that match the pattern index-* (can also be a data stream)

It order to meet Mean Time To Recovery (MTTR) I'd like to have three (3) snapshot policies setup.

  1. SLM_Policy_Hourly_Hot:
    • Hourly snaphot of index-* WHERE index phase is hot
    • S3 Repo: S3_Repo_A
  2. SLM_Policy_Daily_Warm:
    • Daily snapshot of index-* WHERE index phase is warm
    • S3 Repo: S3_Repo_A
  3. SLM_Policy_Daily_Cold:
    • Daily snapshot of index-* WHERE index phase is cold
    • S3 Repo: S3_Repo_B

Is the above currently possible?

2

I don't think you can, but I also don't really understand the advantages of this scheme over simply snapshotting everything hourly into S3_Repo_A and daily into S3_Repo_B. Can you explain your thinking a bit more clearly?

3

Yes, for a bit more clarity, S3_Repo_A has limited storage available to it, and does not have the required space needed to back up everything in index-*. So while policies 1 and 2 could be merged, policy 3 wouldn't be able to. The cold phase will hold the majority of data, and therefore needs to be offloaded to S3_Repo_B.

4

Thanks, I see. I guess we generally expect that every repository is at least as big as the cluster to which it's attached but that isn't the case for you. An interesting idea for sure, but no I don't think it's possible today.

5

Thanks for the information @DavidTurner. Would this make a good feature request for Elasticsearch, or do you think it's something that won't be supported?

6

Seems like a reasonable request, but we're not exactly short of those :wink: It might not happen any time soon, unless it turns out to be a very popular idea.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.