Using Java API multi-thread snapshot index come across concurrent snapshot execution exception

Elastic Stack Elasticsearch
1

Elasticsearch version: 6.6.0

Plugins installed: install snapshot plugin in es cluster.

image
63250048-51d3cd80-c29d-11e9-9e14-e6b3dad41129.png691×286 38.5 KB

JVM version: 1.8.0_131

OS version: Darwin Guobin.local 16.6.0 Darwin Kernel Version 16.6.0: Fri Apr 14 16:21:16 PDT 2017; root:xnu-3789.60.24~6/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

Using Java API multi-thread snapshot index come across ConcurrentSnapshotExecutionException.

Running 3 threads of LogArchiveSnapshotThread-0, LogArchiveSnapshotThread-1 and LogArchiveSnapshotThread-2, snapshot method as below:

public ActionFuture<CreateSnapshotResponse> createSnapshot(String repositoryName, String indexName) {

        String snapshotName = indexName + CommonConstant.SNAPSHOT_SUFFIX;

        ClusterAdminClient clusterAdminClient = elasticsearchUtil.getClusterAdminClient();

        CreateSnapshotRequestBuilder createSnapshotRequestBuilder = clusterAdminClient.prepareCreateSnapshot(repositoryName, snapshotName);

        createSnapshotRequestBuilder.setIndices(indexName)
                .setIncludeGlobalState(false)
                .setWaitForCompletion(true);

        ActionFuture<CreateSnapshotResponse> response = clusterAdminClient.createSnapshot(createSnapshotRequestBuilder.request());

        if (response != null &&
                SnapshotStatusEnum.SUCCESS.name().equals(response.actionGet().getSnapshotInfo().state().name())) {
            logger.info("create snapshot success, current index is " + indexName + ", snapshot is " + snapshotName + ", repository is " + repositoryName);
        }

        return response;
    }

Log info:

2019-08-19 16:14:02.932 ERROR 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { cbp_transfer_webservice-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:cbp_transfer_webservice-2019-08-19_snapshot]  a snapshot is already running]
2019-08-19 16:14:02.932  INFO 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : current thread is LogArchiveSnapshotThread-2, current processing index is remit_ebank_hessian-2019-08-19
2019-08-19 16:14:02.944 ERROR 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { remit_ebank_hessian-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:remit_ebank_hessian-2019-08-19_snapshot]  a snapshot is already running]
2019-08-19 16:14:02.944  INFO 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : current thread is LogArchiveSnapshotThread-2, current processing index is p2f_starter-2019-08-19
2019-08-19 16:14:02.944 ERROR 4278 --- [pool-2-thread-2] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { bankrouter_hessian-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:bankrouter_hessian-2019-08-19_snapshot]  a snapshot is already running]
2019-08-19 16:14:02.944  INFO 4278 --- [pool-2-thread-2] c.y.l.c.common.LogArchiveSnapshotThread  : current thread is LogArchiveSnapshotThread-1, current processing index is monitor_center_hessian-2019-08-19
2019-08-19 16:14:02.965 ERROR 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { p2f_starter-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:p2f_starter-2019-08-19_snapshot]  a snapshot is already running]
2019-08-19 16:14:02.965 ERROR 4278 --- [pool-2-thread-2] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { monitor_center_hessian-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:monitor_center_hessian-2019-08-19_snapshot]  a snapshot is already running]
2019-08-19 16:14:02.965  INFO 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : current thread is LogArchiveSnapshotThread-2, current processing index is pp_cashier_app-2019-08-19
2019-08-19 16:14:02.965  INFO 4278 --- [pool-2-thread-2] c.y.l.c.common.LogArchiveSnapshotThread  : current thread is LogArchiveSnapshotThread-1, current processing index is ap_order_hessian-2019-08-19
2019-08-19 16:14:02.985 ERROR 4278 --- [pool-2-thread-3] c.y.l.c.common.LogArchiveSnapshotThread  : LogArchiveSnapshotThread create snapshot index { pp_cashier_app-2019-08-19 } come across a error, message is ConcurrentSnapshotExecutionException[[test541to660:pp_cashier_app-2019-08-19_snapshot]  a snapshot is already running]

Beside of this problem above, I also want to konw where is the ES java api doc? Thx!!!

2

Is anyone here?

3

please note, that this forum does not come with an SLA. Waiting 15 hours before bumping a thread is pretty early. There is commercial support if you need a guaranteed reply within business hours.

Also, try not to put screenshots in here, they are hard to read and one is unable to copy-paste snippets from screenshots.

Can you explain how you are calling this code? You are not waiting for a snapshot to finish in the code pieces you supplied (as you return the future, not the result), but you can always only create a single snapshot of a repo at the same time.

You also have not mentioned how you are calling this code? Judging from the log messages it seems to be tight loop. Can you explain why?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.