ElasticSearch S3 Repository : Question on what is "repository"?

[titan1978 (Praetor)]

Hello,
I'm trying to understand the nuance of "repository" when it comes to restoring a snapshot on a DIFFERENT newly constructed stack. Here's the scenario:

1. I create an ELK stack in US-EAST-1.
2. I use the PUT/snapshot/my_repository with the { ... } approriate settings to create a respository
3. I create a snapshot successfully in this repository
4. I create an ELK stack in US-WEST-2.
5. In this newly created stack in US-WEST-2 : I first try to get all the snapshots using:
   GET /_snapshot/my_repository/_all
   At this point I get an exception:
   "type": "repository_missing_exception",
   "reason": "[my_repository] missing"

However, if at step 5 BEFORE running that step in this new region, I do the step 2 (Create a repository with the SAME name my_repository using the same PUT API Call) and then attempt the GET all snapshots under the repository - things magically start working!

The Question is :

1. Whenever we create a BRAND NEW ELK stack - is this create repository step mandatory ? If yes - is this like a "pre-registration" of sorts that is mandatory ?
2. How would I account for this in curator if I create a new stack that starts up a curator job (as part of DR where I would need a new stack in a different region) ? would a manual intervention on my part be needed to run this "pre-registration" step needed before curator can do its job?

[warkolm (Mark Walkom)]

1. Yes, an Elasticsearch cluster only knows of the repos you have defined within it's own config.
2. Yes, you want to do that as part of your setup.

Also we’ve renamed ELK to the Elastic Stack, otherwise Beats and APM feel left out! Check out https://www.elastic.co/elk-stack

[titan1978 (Praetor)]

@warkolm Thank you for the quick reply. What exactly is a "repository" in this case(I'm trying to get a picture of what it is tangibly - where is it "stored" on the Elastic Stack physically)

[warkolm (Mark Walkom)]

It's an external thing like an S3 bucket. The only thing Elasticsearch stores is a pointer - the config - to where it can access it.

[titan1978 (Praetor)]

@warkolm thank you! Not to be a pain but could you specify exactly what is "external thing"? Can I look it up or see where its stored - if yes - where? I looked at the s3 bucket where my snapshots are stored and I see no indication of my repository.

[warkolm (Mark Walkom)]

Have a read of https://www.elastic.co/guide/en/elasticsearch/reference/6.2/modules-snapshots.html#_repositories