Snapshot automation sequencing question

Elastic Stack Elasticsearch
1

Hi,

My team is developing c# service to automate snapshot management. It creates and deletes snapshots and writes out a log of existing snapshots and their status, etc. We have a "source" defined for each snapshot, and it includes configs (retention policy, indices to include, other snapshot configs (include cluster state, partial, etc.). Currently we have one source for each of our index types. So, we have source A, for example, for Index type A, and those snapshots will include IndexA_2016.01.01, IndexA_2016.01.02, etc. And a source B for Index type B, etc...1 I have a question about sequencing.

According to your online content, "when a snapshot is deleted from a repository, Elasticsearch deletes all files that are associated with the deleted snapshot and not used by any other snapshots." Considering that, regarding sequencing, I'm thinking:

  1. Delete snapshots before creating snapshots
  2. Delete snapshots starting with the most recent first and working way backward to oldest

Is that correct?

Also, want to confirm that lets say we have these snapshots:

snapshot1
indices included: index1_2016.01.01

snapshot2
indices included: index1_2016.01.01
indices included: index1_2016.01.02

reallyweirdsnapshotname

indices included: index1_2016.01.01
indices included: index1_2016.01.02
indices included: index1_2016.01.03

If I later attempt to restore all indices from reallyweirdsnapshotname, it will pull the files needed from snapshot1 and snapshot2. I don't see how the answer to this could be anything other than yes, but feel compelled to ask all the same.

Thanks,
Casie

2

Hi Casie,

The order in which you delete the snapshots does not matter. Snapshotting saves disk space in snapshots by re-using shared underlying index files (Lucene segments) if they haven't changed. Say for example you have an index idx that you want to snapshot. You take snapshot A of idx when it has the underlying segment files 1, 2, and 3. Those Lucene segment files will get stored as part of the snapshot. Say you indexed a bit more data into A, that generated more Lucene segments 4 and 5. Now if you take snapshot B of idx, it will see that we have already stored files 1, 2, and 3 as part of snapshot A so it won't take a snapshot of those again, instead it will just copy over files 4 and 5 to the snapshot, and have a reference to the already snapshotted files 1, 2, and 3.

Now, suppose you delete snapshot A. The snapshot service will no longer contain snapshot A, but the files snapshotted via A (i.e. 1, 2, and 3 in our example) will still be around and not be deleted because the snapshot service knows that there are other snapshots in the system that need those files (in this case, its snapshot B that references those files).

So it really won't matter what order you delete snapshots in, you won't lose any data. And indeed, when you go to restore B, it will restore all files that B required, even if those files were actually originally snapshotted by A.

3

Thanks for the quick reply!

Regarding this:

"Now, suppose you delete snapshot A. The snapshot service will no longer contain snapshot A, but the files snapshotted via A (i.e. 1, 2, and 3 in our example) will still be around and not be deleted because the snapshot service knows that there are other snapshots in the system that need those files (in this case, its snapshot B that references those files).a'

Let's say we want the segments of A to be deleted. We know if we delete snapshot A, the name snapshot A won't exist anymore, but the segments 1, 2, and 3 still will because they're part of Snapshot B. So that won't work. We should just delete snapshot B, and that would delete all the segments associated to it, regardless of which snapshot originally generated them? If that's the case, would it also delete the reference to snapshot A, assuming there were no segments in it not also associated to Snapshot B?

Thanks,
Casie

4

That's correct, in our example, once you delete snapshot B, then all of the segments will have been deleted (including 1, 2, and 3) because at that point, there are no snapshots in the repository anymore that require those files.

1 Like
5

Regarding your question on references, if you look at the layout of a snapshot repository, you will see various snap-*.dat files in the root, then an indices folder with all the index files, and snapshot metadata files in each of those index directories. The snapshot files in the index directories specify which index files that snapshot requires. Once a file is no longer referenced by any of the snapshot metadata files in the indices folder, it is deleted.

1 Like
6

Here is a link to a blog post that describes how it works quite well.

1 Like
7

It's a great question, and, yes, the order of operations is absolutely significant here.

You should not delete snapshots first. When you delete snapshots, you open the possibility that you remove the only reference to a segment file, and that segment file is deleted. Then, you take a newer snapshot, and if that exact segment file still exists, it will have to be re-streamed from your nodes to the repository. Had you only taken the new snapshot first, there would have been an additional reference to the segment created in the repository, and the segment would have been preserved when you deleted the older snapshot.

Now, for item 2, I'm actually not sure I understand your scenario, but, in general, I would delete the oldest snapshots first.[quote="casieowen, post:3, topic:58740"]
Let's say we want the segments of A to be deleted.
[/quote]

Why, exactly, would you want that?

1 Like
8

Thanks for the suggestion about sequencing (creating first and then deleting).

Re: "Why, exactly, would you want to do that": The idea was not to be consuming space we don't need to consume. But that was before I knew that segments would automatically be deleted once no snapshots no longer reference them. The thought was, if we try to delete old snapshots starting with the oldest first, none of the segments will actually be deleted because there are other newer snapshots referencing those segments. So, if we didn't want to waste processing on deleting snapshots that wouldn't actually delete any segments, then we should start with the newest, but thinking about it more now, it shouldn't matter in which order we delete snapshots we no longer need. We tell elasticsearch which snapshots we don't need more by deleting the snapshot and it handles which segments to delete smartly and the segments will be deleted when they should be deleted (when no snapshots depend on/refer to them). Basically, we don't have to manage it at, or even think about, the segment level.

We'll also be testing this out in depth shortly and will see it for ourselves. Thanks for the insight so far!