Elasticsearch Curator and Amazon S3 filename

Elastic Stack Elasticsearch
1

I have seen the following closed topic: https://discuss.elastic.co/t/curator-snapshot-and-s3-file-name/87185

My use case is similar. Each day I snapshot the previous five days and push them to S3. The plan was to overwrite existing snapshots and fill in any gaps left by failures to ensure a continuous backup. Unfortunately the random naming scheme seemingly makes this impossible.

Perhaps I should be doing it more programmatically?

2

The only way to set and get names for snapshots is via the snapshot API. The file names do not correlate at all.

You also can't "overwrite" existing snapshots. You must understand that a snapshot is actually pointer metadata and segment files. When you initiate a snapshot, you're setting up a sequence of events (roughly this):

  1. Identify all segments used by the indices selected for the snapshot
  2. Stop background segment merges of identified segments
  3. Check the remote repository to see if any of those segments have already been copied
  4. Copy only new segments to the repository
  5. Update the snapshot metadata so that segments in use have a pointer reference.
  6. Report snapshot completion (if successful)
  7. Enable background segment merges of identified segments

This is why a snapshot isn't copying things to a particular file on the remote file system. It's also why you can't "overwrite" a snapshot. The snapshot is technically metadata, and segment files are the actual data.

3

Thanks for clarifying that Aaron. I'll review my approach.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.