Curator 4.2.6 showing Concurrent Snapshot Execution Exception?

Elastic Stack Elasticsearch
1

Hi,
Elasticsearch 2.4.1

My environment has 3 VM's that is hosted on azure and in all the VM's i had installed curator and they are master enabled and in every week i am running 3 snapshots at particular time (9.00AM) . Upto yesterday i didnt shown any error but today i had seeing following ERROR and shocked to see no snapshot has been created.

[2017-06-11 12:00:03,714][WARN ][snapshots                ] [data-1] [backup][snapshot-2017-06-11] failed to create snapshot
ConcurrentSnapshotExecutionException[[backup:snapshot-2017-06-11] a snapshot is already running]
        at org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:209)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2017-06-11 12:00:03,715][WARN ][rest.suppressed          ] path: /_snapshot/backup/snapshot-2017-06-11, params: {repository=backup, wait_for_completion=true, snapshot=snapshot-2017-06-11}
ConcurrentSnapshotExecutionException[[backup:snapshot-2017-06-11] a snapshot is already running]
        at org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:209)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:194)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2017-06-11 12:00:04,169][WARN ][snapshots                ] [data-1] [backup][snapshot-2017-06-11] failed to create snapshot
ConcurrentSnapshotExecutionException[[backup:snapshot-2017-06-11] a snapshot is already running]
        at org.elasticsearch.snapshots.SnapshotsService$1.execute(SnapshotsService.java:209)
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45)
        at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:468)
        at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:772)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:231)
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadP

It is showing snapshot is already running if this is the case then this error should have to error long before when i started running 3 snapshots at the same time but it came yesterday only . Why?

Did wait_for_completion:false will avoid these type of snapshots failure?

Thanks

2

Snapshots cannot run concurrently. Because Elasticsearch must pause segment merging of the segments being snapshotted, and this consumes extra resources and overhead, Elasticsearch does not allow concurrent snapshots. I believe your other question regarding this has been answered elsewhere.

3

Thanks @theuntergeek

Yeah it is :smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.