Restore Snapshot from Deployment-A to Deployment-B

Elastic Stack Elasticsearch
1

Hi there. In Elastic Cloud, I'm trying to do the following:

From my local server (I want to automate it):

  1. issue a curl call of the Elastic API that tells Deployment-A to create a snapshot
  2. then issue a curl call of the Elastic API that tells Deployment-B to restore from the snapshot created in Step 1

Basically, I want the API calls for what happens when you perform the "Restore from another deployment" option in Kibana (attached).

TIA

Screenshot 2024-06-25 122423
Screenshot 2024-06-25 1224231379×377 24 KB

2

Hi,

you can read this docs:

Snapshot and restore | Elasticsearch Guide [8.14] | Elastic

Create snapshot API | Elasticsearch Guide [8.14] | Elastic

Restore snapshot API | Elasticsearch Guide [8.14] | Elastic

Regards

3

Thanks, but I have read those before. They don't address what I described in the screenshots. I have reached out to support, and they admit this is probably some behind-the-scenes magic that they aren't exposing. Waiting for a response.

1 Like
4

I'm also very interested in doing this as well.

I'm looking for an automated way to restore Deployment-B from Deployment-A so I can periodically have my test environment clone the state of Deployment-A

5

Update:

I did some work this week that has almost accomplished what you're looking for here but I have a problem. My steps were:

  1. "GET" "/_snapshot/found-snapshots" from deployment-A's API and store repo_info = response.json()["found-snapshots"]
  2. register the repo on deployment-B as read-only:
@dataclass
class ElasticCluster:
    """Connection details for Elasticsearch API."""

    elasticsearch_host: str
    elasticsearch_api_key: str

# ...
repo_info = copy.deepcopy(repo_info)
# https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html
repo_info["settings"]["readonly"] = True
# repo_info["settings"]["compress"] = False

# where es_request is just a wrapper I made around httpx that adds a header with my API key stored in target_cluster and uses elastic_search_host to update the URL
res = es_request(
    cluster=target_cluster,
    method="PUT",
    path=f"/_snapshot/{target_repo}",
    json=repo_info,
)
  1. Now when I visit /app/management/data/snapshot_restore/repositories/ on Deployment-B's kibana, and try to "Verify repository" on the new repo, I have an error that suggests the S3 client from Deployment-A is not accessible to deployment-B.
"Unknown s3 client name [elastic-internal-XXXXX]. Existing client configs: elastic-internal-YYYYY,default"

Interestingly this isn't a problem on Deployment-C for which I initially used the UI in elastic cloud to restore from a snapshot of Deployment-A. So I think the UI magically granted Deployment-C access to Deployment-A's S3 client. I'm now looking for a way to do this programatically...

  1. The last step I would want to do is hit Deployment-B's API requesting a restore from a desired snapshot in the repo I registered above.

Also btw I started a related thread here in February 2024 Restore from found-snapshots across clusters

6

Final update (sorry to spam this thread but I know somebody will appreciate this being documented in the future):

I was able to get this working by just using the S3 client settings belonging to Deployment-B to configure the read-only snapshot bucket form Deployment A.

# updated code:
def register_repo(
    target_cluster: ElasticCluster,
    target_repo: str,
    remote_repo: Dict,  # result of get_repo_info()
    s3_client: str,  # S3 client inside **target** env to use for reading remote repo
) -> bool:
    """
    Registers the provided snapshot repo, `remote_repo`, inside of `target_cluster` for read (only) access under the name `target_repo`.
    Returns True on success.
    Visit /app/management/data/snapshot_restore/repositories in Kibana to see repos.
    https://www.elastic.co/guide/en/elasticsearch/reference/current/put-snapshot-repo-api.html
    https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html
    """

    remote_repo = copy.deepcopy(remote_repo)
    # https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html
    remote_repo["settings"]["readonly"] = True
    remote_repo["settings"]["client"] = s3_client  # e.g. can't use PRD client in TST
    # repo_info["settings"]["compress"] = False

    res = es_request(
        cluster=target_cluster,
        method="PUT",
        path=f"/_snapshot/{target_repo}",
        json=remote_repo,
    )


def get_repo_info(cluster: ElasticCluster, repo_name: str) -> Optional[Dict]:
    """Get metadata about a snapshot repo e.g. {'type': 's3', 'uuid': ...}"""
    res = es_request(cluster, "GET", f"/_snapshot/{repo_name}")
    if res.status_code != 200:
        logger.warning(
            f"failed to get repo info for '{repo_name}': status {res.status_code}, {res.text}"
        )
        return None
    return res.json()[repo_name]


def es_request(
    cluster: ElasticCluster, method: str, path: str, **httpx_kwargs
) -> httpx.Response:
    """
    Wrapper for making an arbitrary request to Elasticsearch.
    Based on core/app/utils/elasticsearch_http.py but synchronous for simplicity.
    """

    headers = {
        "Authorization": f"ApiKey {cluster.elasticsearch_api_key}",
        "Accept-Encoding": "gzip",  # prefer compressed responses
    }

    es_url = cluster.elasticsearch_host.rstrip("/") + "/" + path.lstrip("/")
    res = httpx.request(method, es_url, headers=headers, **httpx_kwargs)
    return res

This works for my existing environments, but interestingly didn't work for a new environment I made from scratch temporarily to test (which got an AWS access error). This is good enough for me now because my existing environments are all I care about.

The alternative solution to all of this: would theoretically be to create my own S3 bucket in my own aws account, configure it in Deployment-A as a snapshot bucket (via the API) with a snapshot lifecycle plan, configure in my other environments as a read only bucket, and provide credentials to all my deployments so they can read from this bucket (and so Deployment-A can write to it). At least here you'd have full control of the access management of each environment and that wouldn't be so opaque.

And for step 4 I first make an API call that deletes all the indices I'm going to restore, prior to the restore API call.

1 Like
7

Yea, I finally got it to work. Basically I have to create my own S3 bucket on my own AWS account. Then get the master and slave deployments registered to that bucket (slaves must be read-only or you have to delete and re-add them each time). Initiate the snapshot from the master, and then restore from each of the slaves. There were several specific API keys I had to make so that the deployments didn't have too many rights when doing all they needed to do to automate this.

I really wish there was a way to do it automatically like in Kibana, but there is not.

1 Like
8

Thanks for the update and insights, that's more or less my plan as well for the future so nice to know it worked for you :pray:

For now I managed to connect the default "found-snapshots" bucket from my master deployment to the others as read only so that's good enough for now. Connecting brand new deployments seems to not work though so some things are just opaque about how Elasticcloud manages the IAM access rights of separate deployments