Snapshot and restore module (concurrency)

Elastic Stack Elasticsearch
1

As part of the workflow I am creating snapshot for each index (periodically) and backup it in AWS S3 repository (with Elasticsearch plugin).
Those snapshots (one per index) are later on restored in a different cluster that only serves.

Reading the documentation is unclear: from one hand:

Snapshot status A list of currently running snapshots with their
detailed status information can be obtained using the following
command:

GET /_snapshot/_status

which implies that multiple snapshots can run in parallel. From other hand:

The snapshot and restore framework allows running only one snapshot or
one restore operation at a time.

When trying to create more than one snapshot (per index) at a time I'm getting ConcurrentSnapshotExecutionException. My question is why does Elasticsearch doesn't allow to create multiple snapshot (for different indices of course)? It's a big bottleneck in my workflow.
Any information / suggestions for solving this issue / workaround is appreciated.

Edit:

Seperation to multiple repositories will allow me concurrency?

[2017-05-03 01:34:31,059][WARN ][snapshots                ] [es-worker1.use.company.com] [dy_s3_repository_2][s_my_index_name] failed to create snapshot
org.elasticsearch.snapshots.ConcurrentSnapshotExecutionException: [company_s3_repository_1:s_my_index_name] a snapshot is already running
        at org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:185)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:329)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:184)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:154)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

I'm using Elasticsearch 1.4.3 (yea I know, that's old - legacy).
link to SO question.

2

You shouldn’t run concurrent snapshots, even with separate repositories (which I’m not certain would work either). Elasticsearch has to keep those indices being snapshotted in a semi-frozen state, where no segments are being merged until the snapshot is complete. That’s how the snapshot is a guaranteed window in time. Trying to do this across multiple concurrent snapshots could cause all kinds of issues, which is why it isn’t a good idea, if it were permitted (which, again, I’m not certain it is).

It’s also not best practices to do one index per snapshot, especially in a 1.x version. As the snapshot count increases, the number of comparisons to existing segments increases, and it can take an increasingly long time to complete snapshots.

3

I will try to describe my system to be more clear, maybe there is another way to achieve my goal. I have two clusters: the first one only "baking" individual indices: each index is being baked and finally snapshotted to s3. Later on each index is being restored into the second cluster. The problem is in the first cluster: if more than one index is baked and waiting to be snapshotted - he is blocked until the current snapshot will be finished (and indices are not related between them). It's my bottleneck and I need find a solution for this issue. BTW, is there any configuration for speed this process up?

Thanks again

4

That's quite an innovative way of using the snapshot/restore framework that we didn't really think of when it was designed. After all the snapshot/restore framework is a backup system, not cross-cluster replication system. So, it doesn't really support this use case that well. Since we are about to embark on re-design of our snapshot/restore framework, it would be helpful to better understand this use case.

Could you share the motivation for creating such system? Why did you choose to have two clusters instead of a single cluster? Are these clusters geographically distributed? How many indices do you "bake" and how often?

5

Hi Igor,

Thanks for your reply.
We baked our indices in center (geo) and then spread them to different geographies. we have hundreds of indices, each one contains between 100K up to 3M documents (and we soon increase to 15M). each one is baked ~ 10min-2hrs and start over. Each index baking flow is independent.
BTW, any tips to increase performance? just saw max_snapshot_bytes_per_sec. can it help me?

6

Yes, it might. You can also create one cluster per index in the center and snapshot to different repositories (it can be the same bucket, but different base paths).

I think a more sustainable approach would be to just send updates to all your distributed clusters and just index it there.

7

Thanks!
BTW does the snapshots amount has impact on the time? I'm creating each index snapshot in a different folder. Then, periodically, I'm deleting snapshots to keep only few (which also consume expensive time because can't be done in parallel). Does it impact on the speed of snapshot creation? Is it faster to create snapshot if less snapshots exist?

8

How much do you modify indices between snapshots?

9

not much (less than 0.5%), but I'm deleting the index and create all documents (keeping the same _id) quite often (once /twice a day). I have some index folders with 2K-3K snapshots.
Regard the max_snapshot_bytes_per_sec. Can I set it on existing repo or I should create a new one?

10

The snapshot and restore processes are file-based and incremental. That means that if you modify only 0.5% of the index without deleting the index and creating it from scratch we only have to snapshot segments that you created since the last snapshot, which in many cases would be 0.5% of the index. Since we merge the segments from time to time, we need to copy over larger segments occasionally, but even in this case it is typically significantly smaller than copying an entire index. The same is true for the restore process.

However, when you delete the index and reindex all records, the resulting segment files end up to be completely different even if you keep the same ids and content of the record forcing the next snapshot to create a complete copy of the index.

A large number of snapshots slows things down a bit, but in the version 5.x and about this slow down is not very significant (especially comparing to previous versions).

The max_snapshot_bytes_per_sec can be modified by reregistering the same repository.

1 Like
11

@Igor_Motov thanks!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.