Snapshot on fast changing clusters

Elastic Stack Elasticsearch
1

Hey,
So, in the last two weeks i have encountered a problem with one of our clusters.
I have a daily backup procedure running on it and often, while the snapshot procedure is running, we get the following exception:
Indexing cycle failed ElasticsearchStatusException[Elasticsearch exception [type=illegal_argument_exception, reason=Cannot delete indices that are being snapshotted.
This is a rapidly changing cluster, as we override the index often.
Is there a way to automatically enable index deletion and cancelation of the snapshot? Some flag to tell the snapshot to give priority to the index deletion? Or any other way of avoiding the clash of the snapshot and index override?

2

I recall snapshot behaviour having changed lately so it would be useful to know which version of Elasticsearch you are using. I do not know the changes made in detail though so will leave that for someone else.

How do you take snapshots? Are you using the ignore_unavailable parameter?

3

I think the parameter @nadler wants is partial, which all non-EOL versions support. With that parameter, Elasticsearch will let you delete indices while they're being snapshotted, because that flag means that missing shards don't fail the whole snapshot.

4

The ES version is 6.3.2.
I am taking the snapshots via API with the ignore_unavailable set to true.

5

That does sound like the flag i was looking for!
Thanks David, I will try it out.

6

6.3 is EOL, please upgrade :slight_smile: