Unexpected behavior while restoring snapshots from 2.x to 5.x

Elastic Stack Elasticsearch
1

I've tried to restore snapshots from es 2.x to 5.x. I've connected newly created es 5.x to the same S3 repo as es 2.x. But let me point out that only es 2.x has been creating snapshots.

Everything was fine, es 5.x has seen all of the snapshots and restored them.

A few days later I've deleted this es 5.x and tried to repeat the operation to confirm that all runs smoothly. This time, however, newly created es 5.x couldn't see all of the snapshots from S3 repo.

Being more specific, es 5.x cannot see snapshots created after the day of the first trial of snapshots restoring, even though es 2.x see them.

After some googling, I've find out that each major version of elasticsearch should have a separate snapshot repository, but I don't think this is the issue - as I've said, only es 2.x has been creating snapshots (you may say that es 5.x was in implicit readonly mode).

Any ideas what's the cause of this unexpected behavior?

2

I don’t think a repository is in readonly mode if you didn’t explicitly set it.

So once a repository has been created in 5.x, it gets updated.
If you still write in the same repository using a 2.x cluster, then you will get some trouble.

Once upgraded to 5.x, don’t use 2.x cluster anymore.

3

"I don’t think a repository is in readonly mode if you didn’t explicitly set it."
I make snapshots (modify repository) with jenkins' job which fires curator for 2.x cluster only.
5.x didn't have such a job. Does it still mean that 5.x was somehow writing to the repository?

"So once a repository has been created in 5.x, it gets updated."
Updated in such manner that its new entries are invisible for 5.x?
It doesn't make sense to me.

I've separated them now - made a copy of that repo and connected it to 5.x cluster.
But from what you've said, it's for nothing, because original repo has been modified anyway.

Any suggestions what should I do?

4

When you did that:

I've connected newly created es 5.x to the same S3 repo as es 2.x.

It updated the repository files.

Then you did that:

as I've said, only es 2.x has been creating snapshots

IIRC this probably caused some problems. As 2.x has somewhat altered the 5.x files.

But from what you've said, it's for nothing, because original repo has been modified anyway.

May be you need to remove manually some files before recreating the 5.x repository in order to trigger the upgrade operation. But i'm super unsure about this so I prefer pinging @Igor_Motov.

5

That's probably https://github.com/elastic/elasticsearch/issues/21838

1 Like
6

I conclude that if I start doing snapshots with es 5.x on this copied repo, everything will work fine, but I won't be able to restore snapshots made by es 2.x since the update of the repository performed by es 5.x.

Am I wrong? If so, please, correct me and propose a different solution.

I will be able to start testing tomorrow.

7

Theoretically, you are right, but practically I think a better approach would be to start a new repo for 5.x.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.