DELETE snapshot request (which was long-running and had not yet completed) is hung

Elastic Stack Elasticsearch
1

Hi,

I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster with cloud plugin installed, but unfortunately the EC2 access keys configured did not have S3 permissions, so ES was in a weird state, so I sent a DELETE snapshot request and it's stuck for more than a couple of hours, any advice on what to do here to cleanup the snapshot request? Logs don't reveal anything relevant.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/531801394440764%40web11j.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

2

That's strange. Wrong S3 permissions should have caused it to failed
immediately. Could you provide any more details about the permissions, so I
can reproduce it? Meanwhile, restarting the nodes where primary shards of
the stuck index are located is the only option that I can think of.

We are working on improving the performance of snapshot cancelation
(Improve speed of running snapshot cancelation by imotov · Pull Request #5244 · elastic/elasticsearch · GitHub), but it didn't
make it to a release yet.

On Monday, March 10, 2014 4:39:24 AM UTC-4, Swaroop wrote:

Hi,

I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster
with cloud plugin installed, but unfortunately the EC2 access keys
configured did not have S3 permissions, so ES was in a weird state, so I
sent a DELETE snapshot request and it's stuck for more than a couple of
hours, any advice on what to do here to cleanup the snapshot request? Logs
don't reveal anything relevant.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Igor, It seems that the S3 bucket had "PUT only permissions". Regards,

Swaroop 10.03.2014, 17:40, "Igor Motov" <imotov@gmail.com>:

That's strange. Wrong S3 permissions should have caused it to failed immediately. Could you provide any more details about the permissions, so I can reproduce it? Meanwhile, restarting the nodes where primary shards of the stuck index are located is the only option that I can think of. We are working on improving the performance of snapshot cancelation (https://github.com/elasticsearch/elasticsearch/pull/5244), but it didn't make it to a release yet.

On Monday, March 10, 2014 4:39:24 AM UTC-4, Swaroop wrote:

Hi,

I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster with cloud plugin installed, but unfortunately the EC2 access keys configured did not have S3 permissions, so ES was in a weird state, so I sent a DELETE snapshot request and it's stuck for more than a couple of hours, any advice on what to do here to cleanup the snapshot request? Logs don't reveal anything relevant.

Regards,

Swaroop

--

You received this message because you are subscribed to the Google Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "elasticsearch" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1015081394516640%40web27h.yandex.ru.

For more options, visit https://groups.google.com/d/optout.

4

Hi Swaroop, I tried to reproduce it, but in my case snapshot failed
with AccessDenied almost immediately after I started it. Any other pointers
besides PUT only permissions?

On Tuesday, March 11, 2014 1:44:00 AM UTC-4, Swaroop wrote:

Hi Igor,

It seems that the S3 bucket had "PUT only permissions".

Regards,
Swaroop

10.03.2014, 17:40, "Igor Motov" <imo...@gmail.com <javascript:>>:

That's strange. Wrong S3 permissions should have caused it to failed
immediately. Could you provide any more details about the permissions, so I
can reproduce it? Meanwhile, restarting the nodes where primary shards of
the stuck index are located is the only option that I can think of.

We are working on improving the performance of snapshot cancelation (
Improve speed of running snapshot cancelation by imotov · Pull Request #5244 · elastic/elasticsearch · GitHub), but it didn't
make it to a release yet.

On Monday, March 10, 2014 4:39:24 AM UTC-4, Swaroop wrote:

Hi,

I had started a snapshot request on a freshly-indexed ES 1.0.1 cluster
with cloud plugin installed, but unfortunately the EC2 access keys
configured did not have S3 permissions, so ES was in a weird state, so I
sent a DELETE snapshot request and it's stuck for more than a couple of
hours, any advice on what to do here to cleanup the snapshot request? Logs
don't reveal anything relevant.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/c9ef80ee-5512-44bd-b301-54496a31f4b6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5eb27214-a53e-40a6-8ce4-29a312df0b63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.