# Elastic cannot restore incremental snapshot in new cluster

**URL:** <https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052>\
**Category:** Elasticsearch\
**Tags:** snapshot-and-restore\
**Created:** [March 23, 2021, 7:21am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052 "2021-03-23T07:21:33Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 23, 2021, 7:21am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/1 "2021-03-23T07:21:33Z")

</div>

Hello. I have created an s3 repository and created a full snapshot and several incremental snapshots (though the data didn't change since the full snapshot was taken. I'm still working on the development infrastructure). In order to validate the snapshots, I try to restore several indexes on a new cluster. But when I register the repository on the new cluster, it only sees the full snapshot, not the incremental snapshots. Both clusters run Elasticsearch 7.11.1. Do you have any suggestions / ideas on how I can restore the incremental snapshots?  
Later edit: even if the incremental backups contain data that the full backup didn't, they still don't appear

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 23, 2021, 11:23am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/2 "2021-03-23T11:23:21Z")

</div>

What do you mean?

Can you run on both clusters:

```auto
GET /
GET /_cat/snapshots/REPO-NAME-HERE?v

```

And share the outputs here.

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 23, 2021, 4:20pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/3 "2021-03-23T16:20:59Z")

</div>

This is from the cluster where the backup was taken

```auto
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
backup-2021-03-23-h08-m39 SUCCESS 1616481554 06:39:14 1616481615 06:40:15 1m 81 89 0 89
backup-2021-03-23-h08-m52 SUCCESS 1616482370 06:52:50 1616482371 06:52:51 1.2s 81 89 0 89
backup-2021-03-23-h09-m05 SUCCESS 1616483146 07:05:46 1616483147 07:05:47 1s 81 89 0 89
backup-2021-03-23-h09-m08 SUCCESS 1616483294 07:08:14 1616483295 07:08:15 1s 81 89 0 89
backup-2021-03-23-h10-m02 SUCCESS 1616486534 08:02:14 1616486535 08:02:15 1.2s 81 89 0 89
backup-2021-03-23-h10-m04 SUCCESS 1616486685 08:04:45 1616486686 08:04:46 1.2s 82 90 0 90

```

This is from the cluster where I am trying to restore

```auto
id status start_epoch start_time end_epoch end_time duration indices successful_shards failed_shards total_shards
backup-2021-03-23-h08-m39 SUCCESS 1616481554 06:39:14 1616481615 06:40:15 1m 81 89 0 89

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 23, 2021, 4:57pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/4 "2021-03-23T16:57:01Z")

</div>

I do not see what is wrong here. All snapshots are full snapshots, although data is reused between them which is sometimes referred to as the incremental part. What were you expecting to see?

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 23, 2021, 4:57pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/5 "2021-03-23T16:57:48Z")

</div>

I expected to see all backups from the first cluster appear on the second cluster

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 23, 2021, 4:59pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/6 "2021-03-23T16:59:14Z")

</div>

I am not sure I understand. As all backups are full you can only restore one.

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 23, 2021, 5:01pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/7 "2021-03-23T17:01:05Z")

</div>

Isn't there a mechanism that if I restore backup `backup-2021-03-23-h10-m04` on the new cluster, the other backups are restored before him? I can only restore to the first backup taken?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 23, 2021, 5:47pm UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/8 "2021-03-23T17:47:35Z")

</div>

All backups are **full** and contain the data in the cluster at the time it was taken. If segments have not changed between snapshots they are reused (snapshots keep track which segments are used by how many snapshots), which means a full copy of all data is not added to the repository for every snapshot - only segments that are new are added, which is where the incremental description comes from. You should probably always restore the latest snapshot only unless you want to go back to a previous point in time.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 24, 2021, 3:12am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/9 "2021-03-24T03:12:29Z")

</div>

You did not share the output of

```
GET /

```

On both clusters. Could you?

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 24, 2021, 5:54am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/10 "2021-03-24T05:54:37Z")

</div>

This is from the first cluster

```auto
{
  "name" : "...",
  "cluster_name" : "es-storage-backup",
  "cluster_uuid" : "...",
  "version" : {
    "number" : "7.11.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a",
    "build_date" : "2021-02-15T13:44:09.394032Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

This is from the second cluster

```auto
{
  "name" : "...",
  "cluster_name" : "es-staging-restore",
  "cluster_uuid" : "...",
  "version" : {
    "number" : "7.11.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a",
    "build_date" : "2021-02-15T13:44:09.394032Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 24, 2021, 9:10am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/11 "2021-03-24T09:10:36Z")

</div>

Could you share the repository settings from both clusters?

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 25, 2021, 8:45am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/12 "2021-03-25T08:45:16Z")

</div>

I think I found the issue. So, the backup runs once, registers the repository on the test cluster, does a first restore and all goes ok. On the second run, it creates a new snapshot on the same repository, then on the test cluster it sends the create repository command, which now does nothing and doesn't update the list of known snapshots on the repository.  
I'm not sure if it's a bug or FAD.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [March 25, 2021, 9:04am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/13 "2021-03-25T09:04:04Z")

</div>

> [@Adrian\_Birladeanu](#):
>
> doesn't update the list of known snapshots on the repository

I think this means the repository in the new cluster is not registered with `readonly: true`, which is very dangerous. You must not register the same repository as writeable on two different clusters.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 25, 2021, 9:08am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/14 "2021-03-25T09:08:05Z")

</div>

> [@DavidTurner](#):
>
> You must not register the same repository as writeable on two different clusters.

I think I was not myself aware of this 😛  
I thought it was something ok as long as the exact same version is used in both clusters.

I guess I should read [the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html#snapshots-read-only-repository). 🤦‍♂️

> If you register the same snapshot repository with multiple clusters, only one cluster should have write access to the repository. Having multiple clusters write to the repository at the same time risks corrupting the contents of the repository.

---

<div class="post-metadata">

**Author:** ![Adrian\_Birladeanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrian_birladeanu/32/63195_2.png) [@Adrian\_Birladeanu](https://discuss.elastic.co/u/Adrian_Birladeanu)\
**Post date:** [March 25, 2021, 9:41am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/15 "2021-03-25T09:41:27Z")

</div>

Thank you very much. I just tried to register the repository on the second cluster as readonly and it reloads the snapshots automatically.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2021, 9:42am UTC](https://discuss.elastic.co/t/elastic-cannot-restore-incremental-snapshot-in-new-cluster/268052/16 "2021-04-22T09:42:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.