Elasticsearch delete snapshot, 504 timeout

Elastic Stack Elasticsearch
#1

Hello,
I am taking snapshots and saving these to AWS S3, when I try to delete old snapshots, I get

curl -X DELETE "http://localhost:9300/_snapshot/backup-prod/20200123031645?pretty"                        ✔  10089  09:20:30 
<html>
<head><title>504 Gateway Time-out</title></head>
<body>
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>

i see there is no wait_for_completion available for this endpoint - Task management API | Elasticsearch Reference [7.11] | Elastic

any advice on how to resolve this, is much appreciated.

#2

Elasticsearch is not returning this error, it's coming from something else in your system applying an unnecessary timeout. The best fix is to remove that timeout. Alternatively you can ignore the error, since the operation will continue in the background. If you like, you can retry it too.

#3

When retrying I get an error:

cannot delete - another snapshot is currently being deleted

So just to confirm, I have quite a lot old snapshots that I want to delete, so when sending

curl -X DELETE "http://localhost:9300/_snapshot/backup-prod/20200123*?pretty"

command, i can ignore the timeout error, but as snapshot status endpoint only returns an object when there is a snapshot being taken or if it's being restored.

it's empty if the snapshot is being deleted and there is no other endpoint to track snapshot deletion operations specifically.

how do i track the snapshot delete progress?

#4

Just to reiterate, the real problem is a bad timeout somewhere else in your system, which you can't fix within Elasticsearch. I strongly recommend addressing that rather than trying to work around it.

This message is also not coming from Elasticsearch, and Elasticsearch 7.11 will let you try and delete the same snapshot twice. You should be able to retry the delete until it either succeeds or indicates that the snapshot was not found.

#5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.