Same Snap Repository, Less Snaps

Using AWS ES Service, with repository-s3 plugin, I have two instances, launched with the same settings. I register the same snap repo successfully, but when I GET /_cat/snapshots/repo_name?v the second instance (created after the first for testing purposes) only retrieves snaps up to a certain date. GET /_snapshots/repo_name/snapname_missing also confirms this.

Where would I begin to look to troubleshoot this difference? I realize Elastic Co doesn't support AWS service, but I'm stumped. And I think this is an issue that has intermittently affected restores in the production instance as well (restore op accepted, but restored to red, due to an S3 404)...

Do you mean you have two clusters both taking snapshots into the same snapshot repository? If so, that will definitely lead to lost data. See these docs:

If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. All other clusters connected to that repository should set the repository to readonly mode.

That's true. And specifically in the near future that might not be the same source code.

Did you look at Cloud by Elastic, also available if needed from AWS Marketplace?

Cloud by elastic is one way to have access to all features, all managed by us. Think about what is there yet like Security, Monitoring, Reporting, SQL, Canvas, Maps UI, Alerting and built-in solutions named Observability, Security, Enterprise Search and what is coming next :slight_smile: ...

1 Like

So I have a staging container cluster, and a production service cluster. The production service only ever restores from the snap repo, the staging container writes and reads. Both are at version 7.7.0. But I didn't explicitly set the production service to read-only from the repository, is that still an issue?

Yea, the AWS service is quite disappointing. I'm a developer for a small startup, our choice to use AWS is due to a pile of credits given to us. I was thinking about switching to an ECS solution, so at least I would be running Elastic official. :+1:


1 Like

Appreciate your help, this forum has been excellent.

If I were to change the snapshot settings on the production service to read-only now, will there be any residual effect? Or will I have to re-create the repository to avoid future issues?

Hard to say for sure, the safest option is definitely to make a new repo. If you are certain it never wrote to the repo then you might be ok (although that reasoning might not hold in future versions).

1 Like