Snapshot in progress for long time. How do I delete it?

Elastic Stack Elasticsearch
1

Hi

We've been using the snapshot feature for a while. Yesterday we hit the
endpoint to create a snapshot however it has been in progress for nearly 24
hours now - this has never happened they generally tend to finish quite
quick. We can't seem to kill it having tried to delete it via the snapshot
api. Our main concern right now is to kill the snapshot and take another
one just so we can back it up. Any help appreciated.

We are using Elasticsearch 1.1.1 running on centos6, java7. We currently
have 6 data nodes and 3 eligible master nodes, one of which is the master.

Dip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/844966ed-a4da-4595-95f9-e80e45343603%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Bit more info. We are using the AWS plugin to write to an s3 location. In
this location I can see the metadata for this snapshot but it's permanently
saying 'IN_PROGRESS'

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3b328fac-5821-4bb7-aef1-06a92595b41c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

I think you can safely delete the snapshot. It will cancel the
still-running process.

From their docs (

):
"When a snapshot is deleted from a repository, Elasticsearch deletes all
files that are associated with the deleted snapshot and not used by any
other snapshots. If the deleted snapshot operation is executed while the
snapshot is being created the snapshotting process will be aborted and all
files created as part of the snapshotting process will be cleaned.
Therefore, the delete snapshot operation can be used to cancel long running
snapshot operations that were started by mistake."

Em sexta-feira, 25 de abril de 2014 10h21min35s UTC-3, Dipesh Patel
escreveu:

Hi

We've been using the snapshot feature for a while. Yesterday we hit the
endpoint to create a snapshot however it has been in progress for nearly 24
hours now - this has never happened they generally tend to finish quite
quick. We can't seem to kill it having tried to delete it via the snapshot
api. Our main concern right now is to kill the snapshot and take another
one just so we can back it up. Any help appreciated.

We are using Elasticsearch 1.1.1 running on centos6, java7. We currently
have 6 data nodes and 3 eligible master nodes, one of which is the master.

Dip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5c9b2899-d289-4d67-8ce4-a6c00456f673%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hi

We've tried using the delete api but it just hangs and doesn't seem to
complete. There isn't anything useful in the log messages.

Dip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8780ba8e-75bc-4419-abc7-76a0abe4c521%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Bit more info, the cluster state api is listing the troublesome snapshot as
being 'ABORTED'. So localhost:9200:_cluster/state:
...

"snapshots": {"snapshots": [{
"repository": "s3_repository",
"snapshot": "custom_snapshot_24_04_2014_17:27"
"include_global_state": true,
"state": "ABORTED",
"indices": [
...

On Monday, April 28, 2014 10:04:16 AM UTC+1, Dipesh Patel wrote:

Hi

We've tried using the delete api but it just hangs and doesn't seem to
complete. There isn't anything useful in the log messages.

Dip

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/96646385-5d4d-4f09-9146-8d1287ddabc8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

hi Dipesh, we are facing the same issue. How did you overcome it (if at all) ?

7

Rolling restart as suggested in Snapshot operation stuck IN_PROGRESS, delete command doesn't work solved the issue.