Will the snapshot repository able to capture the changes in mappings?

[ian.chan]

Hi.

Let's say I have registered a snapshot repository for an index A, with slm policy taking snapshot every hour.
Then I add a new field in the mapping of the index A, and add some new documents with values in this new field.
My questions are:

1. Will the snapshot repository able to capture the changes?
2. If I have another index named B, that have the same mapping as index A before the above changes, and also restored data using the snapshots in the snapshot repository before the above changes. Now after the above changes, can I restore the data into index B using the snapshot repository, given that index B is still using the old mapping?
3. Continue on the discussion of point 2, if I first manually update the mapping of Index B, can I now restore the data of the latest snapshot into index B?
4. If I create a new index C, using the latest mapping of index A, can I restore the data into this index C using the snapshot repository?

Many Thanks!

[Opster_support (Elasticsearch community support @ Opster)]

1. Yes, the snapshot repository will capture the changes. Snapshots in Elasticsearch and OpenSearch are incremental. This means that each new snapshot only stores data that has changed since the last successful snapshot. When you add a new field to the mapping of an index and add new documents, these changes will be included in the next snapshot.

2. No, you cannot restore the data into index B using the snapshot repository if index B is still using the old mapping. The reason is that the snapshot restore process does not allow for the mapping of the index to be changed. The mapping of the index at the time of the snapshot is what will be restored. If the mapping of the index has changed since the snapshot was taken, the restore process will fail.

3. Yes, if you first manually update the mapping of Index B to match the mapping of the snapshot you want to restore, you can now restore the data of the latest snapshot into index B. The mapping of the index at the time of the restore needs to match the mapping of the index at the time of the snapshot.

4. Yes, if you create a new index C, using the latest mapping of index A, you can restore the data into this index C using the snapshot repository. The mapping of the new index C needs to match the mapping of the index at the time of the snapshot. The restore process will then restore the data into the new index C.

Remember, before performing any operations, it's always a good idea to test them in a non-production environment to ensure they work as expected.

[system]

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[Christian_Dahlqvist]

@ian.chan

• If I have another index named B, that have the same mapping as index A before the above changes, and also restored data using the snapshots in the snapshot repository before the above changes. Now after the above changes, can I restore the data into index B using the snapshot repository, given that index B is still using the old mapping?
• Continue on the discussion of point 2, if I first manually update the mapping of Index B, can I now restore the data of the latest snapshot into index B?
• If I create a new index C, using the latest mapping of index A, can I restore the data into this index C using the snapshot repository?

The restore API allows you to restore an index from a snaphot. This will be restored with the mappings that existed when the snapshot was taken. The index can be restored into its original name or be restored using a different name. The restore process does however NOT allow you to merge data from a snapshotted index into another index, so the answer to questions 2, 3 and 4 is No.

@Opster_support

Snapshots in Elasticsearch and OpenSearch are incremental. This means that each new snapshot only stores data that has changed since the last successful snapshot.

Notev that this is not strictly true. Snapshots work at the segment level and not at the data level. If a segment within a shard has not changed between snapshots, it will be reused and not copied multiple times. If segments have merged it is however possible that the snapshot repository will contain multiple segments holding the same data.

[DavidTurner]

@Opster_support

Snapshots in Elasticsearch and OpenSearch are incremental. This means that each new snapshot only stores data that has changed since the last successful snapshot.

This is not correct. Snapshots in Elasticsearch are deduplicated, but logically independent, quite unlike incremental backups which depend on earlier backups.

@Opster_support

No, you cannot restore the data into index B using the snapshot repository if index B is still using the old mapping. The reason is that the snapshot restore process does not allow for the mapping of the index to be changed.

This is not correct. When you restore an index, you restore its mapping too.

@Opster_support

Yes, if you first manually update the mapping of Index B to match the mapping of the snapshot you want to restore, you can now restore the data of the latest snapshot into index B.

This is also not correct, for the same reason. No need to manually update any mappings.

@Opster_support

The mapping of the new index C needs to match the mapping of the index at the time of the snapshot. The restore process will then restore the data into the new index C.

This is also not correct, for the same reason. The restore process restores both data and mappings.

Was this answer partially written by OpsGPT like some of your earlier answers? It's very nicely written, but fundamentally incorrect.

[ian.chan]

Hi.

Thank you for the explanation.

Sound like the restore api will , if necessary, replace all the existing data and mapping of the index I am restoring to using the snapshot, did I get you right?

Also, if the mapping of the snapshot or the mapping of the index I am restoring to had been set to {"dynamic":"strict"}, will it affect the restore?

Many Thanks!

[Christian_Dahlqvist]

If you want to restore an index that still exists in the cluster you will need to either first delete it and then restore it to the state of when the snapshot was taken or restore the index as a new index under a new name.

When you restore the mapping present at the time when the snapshot was taken is restored. Whether the mapping is strict or not has no impact on this.