# Snapshot exception

**URL:** <https://discuss.elastic.co/t/snapshot-exception/251861>\
**Category:** Elasticsearch\
**Created:** [October 13, 2020, 7:48am UTC](https://discuss.elastic.co/t/snapshot-exception/251861 "2020-10-13T07:48:06Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 13, 2020, 7:48am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/1 "2020-10-13T07:48:06Z")

</div>

Hi

i'm trying to take snapshot for every 30 min ,  
The following problems occur.  
ERROR)  
{  
"type": "concurrent\_snapshot\_execution\_exception",  
"reason": "[reponm-prd-snapshots:scheduled-lm\_rkid\_qtoa75fr0qvlca] a snapshot is already running",  
"stack\_trace": "ConcurrentSnapshotExecutionException[[reponm-prd-snapshots:scheduled-lm\_rkid\_qtoa75fr0qvlca] a snapshot is already running]\n\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:203)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"  
}

```
The spec are the same.
elasticsearch 7.8.0
total node : 3 ea(master/data , master/data, master/data) clustering
{
  "nodes": {
    "sqMr-H3dRd6zqtcVysB-xg": {
      "name": "node-3"
    },
    "EvHTgdmkSF-JnTwhORAV-Q": {
      "name": "node-1"
    },
    "AUi5OleeTiKDGDhtYosJpw": {
      "name": "node-2"
    }
  }
}

```

I don't know what's wrong.  
please help me !!

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2020, 9:20am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/2 "2020-10-13T09:20:34Z")

</div>

Welcome.

Please [read this](https://discuss.elastic.co/t/about-the-elasticsearch-category/21) about how to format.

Here the error message seems to be clear:

> a snapshot is already running

You should probably check before running the new snapshot that the previous one has finished. Or just "ignore" the error message and try again 30 minutes later.

---

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 14, 2020, 1:05am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/3 "2020-10-14T01:05:54Z")

</div>

- Execution by schedule =\> Error  
The current problem is the same problem even if the time interval is increased.  
And snapshot execution is executed only by schedule.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 14, 2020, 5:00am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/4 "2020-10-14T05:00:35Z")

</div>

I don't understand. Could you clarify?

---

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 14, 2020, 6:28am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/5 "2020-10-14T06:28:29Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da1e6126ec3992bb4e13f7532b1b0a5a8ea609a6.png)

I registered the schedule in snapshot &restore,  
I just ran the processor according to the schedule.  
But I get the same error as above.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 14, 2020, 6:41am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/6 "2020-10-14T06:41:48Z")

</div>

If you want to run the snapshot every 30 minutes, should the cron schedule not look something like this `0,30 * * * *` ? Note that if the snapshot does not complete within 30 minutes you are likely to see the same problem with this schedule.

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [October 14, 2020, 7:35am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/7 "2020-10-14T07:35:14Z")

</div>

Hi @pmk

one thing to add to what @Christian_Dahlqvist points out

> Note that if the snapshot does not complete within 30 minutes you are likely to see the same \> problem with this schedule.

This is a non-issue if you were to upgrade to v7.9 or later. We support fully concurrent snapshot operations from that version on. See:

> <https://github.com/elastic/elasticsearch/pull/56911>
>
> Enables fully concurrent snapshot operations:
> \* Snapshot create- and delete ope…rations can be started in any order
> \* Delete operations wait for snapshot finalization to finish, are batched as much as possible to improve efficiency and once enqueued in the cluster state prevent new snapshots from starting on data nodes until executed
> \* We could be even more concurrent here in a follow-up by interleaving deletes and snapshots on a per-shard level. I decided not to do this for now since it seemed not worth the added complexity yet. Due to batching+deduplicating of deletes the pain of having a delete stuck behind a long -running snapshot seemed manageable (dropped client connections + resulting retries don't cause issues due to deduplication of delete jobs, batching of deletes allows enqueuing more and more deletes even if a snapshot blocks for a long time that will all be executed in essentially constant time (due to bulk snapshot deletion, deleting multiple snapshots is mostly about as fast as deleting a single one))
> \* Snapshot creation is completely concurrent across shards, but per shard snapshots are linearized for each repository as are snapshot finalizations
> 
> See updated JavaDoc and added test cases for more details and illustration on the functionality.
> 
> Some notes:
> 
> The queuing of snapshot finalizations and deletes and the related locking/synchronization is a little awkward in this version but can be much simplified with some refactoring. The problem is that snapshot finalizations resolve their listeners on the \`SNAPSHOT\` pool while deletes resolve the listener on the master update thread. With some refactoring both of these could be moved to the master update thread, effectively removing the need for any synchronization around the \`SnapshotService\` state. I didn't do this refactoring here because it's a fairly large change and not necessary for the functionality but plan to do so in a follow-up.
> 
> This change allows for completely removing any trickery around synchronizing deletes and snapshots from SLM and 100% does away with SLM errors from collisions between deletes and snapshots.
> 
> Snapshotting a single index in parallel to a long running full backup will execute without having to wait for the long running backup as required by the ILM/SLM use case of moving indices to "snapshot tier". Finalizations are linearized but ordered according to which snapshot saw all of its shards complete first

---

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 14, 2020, 7:52am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/8 "2020-10-14T07:52:30Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4f56042714d17eefd15efc9360aaefb670bbf6e2.png)  
duration : 170s  
The schedule is changed from 30 min every day to once a day  
But ERROR message is output

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [October 14, 2020, 7:58am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/9 "2020-10-14T07:58:15Z")

</div>

@pmk

> But ERROR message is output

That is strange but I think there might be a bug here, we've had similar but never properly reproduced reports before.

Could you paste the logs around the error message maybe, including the part where the `SnapshotsService` logs the start of the snapshot that actually works out so I can take a look?

Thanks!

---

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 14, 2020, 8:02am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/10 "2020-10-14T08:02:44Z")

</div>

setting

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/6/9640a65f7310a07cb61e3a51298f5c91888d6398.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/b/5b7b6b591978a7868fa1d0e83b98722cc7a48d8e.png)

error message)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/b/eb4a788876d45f4ea72ff00792d735537c85c711.png)

```auto
        {
          "type": "concurrent_snapshot_execution_exception",
          "reason": "[reponm-prd-snapshots:scheduled-v6zsv9rxrhgazmcdhrwlig] a snapshot is already running",
          "stack_trace": "ConcurrentSnapshotExecutionException[[reponm-prd-snapshots:scheduled-v6zsv9rxrhgazmcdhrwlig] a snapshot is already running]\n\tat org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:203)\n\tat org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:47)\n\tat org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:702)\n\tat org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:324)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:219)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n"
        }

```

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [October 15, 2020, 8:09am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/11 "2020-10-15T08:09:21Z")

</div>

@pmk sorry I should have worded this more carefully.

What I'm looking for is full logs with timestamps that show both the `ERROR` for failing to start a snapshot but also the logs for when the concurrent snapshot that prevented the failing one to start happened. So everything between a line like this for the running snapshot:

```auto
[2020-10-15T10:06:51,149][INFO][o.e.s.SnapshotsService] [node_s0] snapshot [test-repo:test-snap/88ZwRkUERZClvs2_0w4DQA] started

```

and it's corresponding completion log which looks like this:

```auto
[2020-10-15T10:06:51,574][INFO][o.e.s.SnapshotsService] [node_s0] snapshot [test-repo:test-snap/88ZwRkUERZClvs2_0w4DQA] completed with state [SUCCESS]

```

would be ideal if possible.

Thanks again!

---

<div class="post-metadata">

**Author:** ![pmk](https://avatars.discourse-cdn.com/v4/letter/p/97f17d/32.png) [@pmk](https://discuss.elastic.co/u/pmk)\
**Post date:** [October 15, 2020, 8:51am UTC](https://discuss.elastic.co/t/snapshot-exception/251861/12 "2020-10-15T08:51:15Z")

</div>

Due to security policy, files cannot be attached.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3ac71d3a10606216230c6c5ffa0b2f5ffac870ee.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d9da2dce8768bf5c9b8c28d851963ca3db22f21c.png)

---

<div class="post-metadata">

**Author:** ![Armin\_Braun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armin_braun/32/20092_2.png) [@Armin\_Braun](https://discuss.elastic.co/u/Armin_Braun)\
**Post date:** [October 15, 2020, 3:41pm UTC](https://discuss.elastic.co/t/snapshot-exception/251861/13 "2020-10-15T15:41:25Z")

</div>

Thanks @pmk

we observed this issue in another context as well today and think this is a bug. We're tracking the work on it in the below issue now

> <https://github.com/elastic/elasticsearch/issues/63754>
>
> Elasticsearch version (bin/elasticsearch --version): 7.9
> Description of the problem including expected versus actual behavior: A SLM policy is getting triggered twice at...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 12, 2020, 3:41pm UTC](https://discuss.elastic.co/t/snapshot-exception/251861/14 "2020-11-12T15:41:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.