Curator Snapshot and S3 file name

Elastic Stack Elasticsearch
1

I have curator configured and working properly by taking snapshots and sending them to a bucket in S3. What I can't seem to understand is how to have the filename in S3 match the snapshot name specified in the config file which I can clearly see when I run:
'_cat/snapshots/s3_repo?v'

the name of the snapshot is the one specified in my config file, however when I look at my s3 bucket, the filename for the .dat file appears to be saved as meta- followed by a random string. Is there some setting I need to add in the curator config to specify the name of the output file and not just the name of the snapshot?

2

This is how newer versions of Elasticsearch do it. Indices are also named similarly. The snapshot name is only in the metadata, accessible via API call.

3

Aaron,
Thanks for the info and was wondering whether you can expand a bit more on that as I'm trying to understand exactly what files I'm supposed to be seeing in the S3 bucket for a successful snapshot. I see files with meta-random_string and one with snap-random_string and there's also an index-(number) and I also believe there's an indices folder ... So, just wanting to have a very clear picture of what a typical snapshot should generate. Thanks for the help.

4

You're better off using the API to validate that snapshots were correctly taken. A successful snapshot will have state: SUCCESS.

GET /_snapshot/REPO_NAME/_all

However, Curator provides a CLI tool and a filter to check the state of snapshots:

curator_cli --host 127.0.0.1 --port 9200 show_snapshots --repository REPO_NAME --filter_list '[{"filtertype":"state","state":"SUCCESS"}]'

If you have a client config file in ~/.curator/curator.yml then curator_cli will auto detect it, and all you would need to run is:

curator_cli show_snapshots --repository REPO_NAME --filter_list '[{"filtertype":"state","state":"SUCCESS"}]'

or you can point it to the config file with --config

curator_cli --config /path/to/curator_config.yml show_snapshots --repository REPO_NAME --filter_list '[{"filtertype":"state","state":"SUCCESS"}]'

What will output is a list of all snapshots which were successful.

5

Aaron,
Yeah I ran all that after reading the docs and as previously stated, I verified that the snapshot succeeded by running some of the commands you mention. However, I'm just curious to know more about the structure of the files generated with a successful snapshot as I haven't come across much on that.

6

They are segments, and files that help organize them and catalog them so Elasticsearch can easily see what's there.

We recommend these files never be touched directly, so you won't find any documentation about them. If you really want to know what's up there, you will have to peruse the code. I won't be making any recommendations other than what I have.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.