Taking snapshots concurrently of each index in Elasticsearch

Hi!

I have written a module which issues snapshot creation requests for an index and sends it off to a ES cluster. This module is deployed on ECS and can run issue snapshot creation requests concurrently.

Right now, The snapshot creation request looks like,

resp, err := esclient.SnapshotCreate(esRepository, snapshotName).
	WaitForCompletion(true).BodyJson(map[string]interface{}{
	"indices": "indexname",
	"metadata": map[string]string{
		"taken_by": "native_snapshot_ecs_task",
	},
}).Do(context.Background())
if err != nil {
	return time.Since(startTime), err
}

If ES cluster receives multiple requests in parallel, It goes ahead with the one that came first and returns a illegal_state_exception error. (Error 500 (Internal Server Error): trying to modify or unregister repository that is currently used [type=illegal_state_exception]).

Since, I am always only taking snapshot of 1 index/request and no two snapshot creation requests share an index, What can I do to make this all work?

I know there is, include_global_state setting but I am not sure if that is what I should be using?

Hi @ishanjain

I'm afraid nothing. Parallel snapshot operations are currently simply not supported in a cluster without exception. This might change in the mid-term but for now you are limited to a single snapshot at a time per cluster, unfortunately.

Okay, Thanks. We'll figure something out so we don't run in this situation.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.