ES5 cannot find latest ES2 snapshot

[Dvine]

Hi,

We are migrating to an ES5.4 cluster from an ES2.4.3 Cluster. Snap shots are in S3 and we want to restore from the ES2.4 repo to our new cluster. We use creator to restore this and take our daily snapshots.

The problem we have is that curator and http://ES5cluster:9200/_snapshot/reponame/_all is only finding a latest snapshot from almost 6 months ago. http://ES2cluster:9200/_snapshot/reponame/_all successfully finds all the snapshots since last nights successful back up.

If I CURL the correct snapshot from last night ES5 claims it cannot find it. The last ES5 http://ES5cluster:9200/_snapshot/reponame/_all returns this one.

{"snapshot":"snapshot-20170116010510","uuid":"snapshot-20170116010510","version_id":2040399,"version":"2.4.3","indices":[ <ignored for size],"state":"SUCCESS","start_time":"2017-01-16T01:05:25.737Z","start_time_in_millis":1484528725737,"end_time":"2017-01-16T01:24:30.814Z","end_time_in_millis":1484529870814,"duration_in_millis":1145077,"failures":[],"shards":{"total":5822,"failed":0,"successful":5822}}

The last ES2 looks like this:

{"snapshot":"snapshot-20170530010505","version_id":2040399,"version":"2.4.3","indices":[ <index list> ],"state":"SUCCESS","start_time":"2017-05-30T01:06:10.523Z","start_time_in_millis":1496106370523,"end_time":"2017-05-30T01:18:46.628Z","end_time_in_millis":1496107126628,"duration_in_millis":756105,"failures":[],"shards":{"total":8336,"failed":0,"successful":8336}}

Any reason why this is? Both look to be successful snapshots.

[dadoonet (David Pilato)]

Once you have migrated a repository to a 5.x cluster, you can't and you should not use it anymore with a 2.x cluster.

That might causes issues.

When a repository is created in 5.x, meta data files are updated. Might explain what you are seeing.

[Dvine]

The ES5 cluster has a new repository for its snapshots and only uses the old ES2 repo to restore. Would that update the metafiles? And if so why has it only affected the last 6 months (actually 5.5 months) worth or snapshots? As I said the ES2 cluster has no problem using its own snapshot from last night so I don't think the ES5 cluster has changed anything here.

The ES5 cluster can literally not see any snapshots from the middle of Jan (its only been running the last 2 days).

[dadoonet (David Pilato)]

If I read correctly your answer, you are still have connected a 2.x cluster on the same repository which is used by the 5.x cluster.

This is not supported. Read Snapshot And Restore | Elasticsearch Reference [5.4] | Elastic

Note: If a repository is connected to a 2.x cluster, and you want to connect a 5.x cluster to the same repository, you will have to either first set the 2.x repository to readonly mode (see below for details on readonly mode) or create the 5.x repository in readonly mode. A 5.x cluster will update the repository to conform to 5.x specific formats, which will mean that any new snapshots written via the 2.x cluster will not be visible to the 5.x cluster, and vice versa. In fact, as a general rule, only one cluster should connect to the same repository location with write access; all other clusters connected to the same repository should be set to readonly mode. While setting all but one repositories to readonly should work with multiple clusters differing by one major version, it is not a supported configuration.

[Dvine]

More I have connected a 5.x cluster to a 2.x repo. And this seemed to be the cause.

I created a new repo, backed up to that and then managed to restore correctly. Thanks for your help.