Snapshot is stuck in IN_PROGRESS

Elastic Stack Elasticsearch
1

I am using cloud aws plugin, Snapshots are taken directly to S3.
It's a 3 node cluster.

ES config:
cloud:
aws:
access_key: xxx
secret_key: yyy
protocol: http
region: us-west-1
discovery.type: ec2
discovery.ec2.groups: elasticsearch
repositories:
s3:
bucket: "xxx"
region: "us-west-1"

When I run this

curl -XGET
'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'

Part of the output is
"snapshots" : [ {
"snapshot" : "2015.03.11.06.04.39",
"repository" : "my_s3_repository",
"state" : "ABORTED",
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 17,
"failed" : 3,
"total" : 20
},

When I run

curl localhost:9200/_snapshot/_all?pretty

it shows IN_PROGRESS
{
"snapshot" : "2015.03.11.06.04.39",
"indices" : [ "playground_video", "playground_admin",
"playground_profile", "playground_feed" ],
"state" : "IN_PROGRESS",
"start_time" : "2015-03-11T06:04:39.044Z",
"start_time_in_millis" : 1426053879044,
"failures" : ,
"shards" : {
"total" : 0,
"failed" : 0,
"successful" : 0
}
}

curl -XDELETE

'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'

this operation never completes. I left it running for a day..still no use..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

What version are you on?

You can try deleting the snapshot and trying again.

On 1 April 2015 at 21:53, Pradeep Reddy pradeepreddy.manu.iitkgp@gmail.com
wrote:

I am using cloud aws plugin, Snapshots are taken directly to S3.
It's a 3 node cluster.

ES config:
cloud:
aws:
access_key: xxx
secret_key: yyy
protocol: http
region: us-west-1
discovery.type: ec2
discovery.ec2.groups: elasticsearch
repositories:
s3:
bucket: "xxx"
region: "us-west-1"

When I run this

curl -XGET
'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'

Part of the output is
"snapshots" : [ {
"snapshot" : "2015.03.11.06.04.39",
"repository" : "my_s3_repository",
"state" : "ABORTED",
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 17,
"failed" : 3,
"total" : 20
},

When I run

curl localhost:9200/_snapshot/_all?pretty

it shows IN_PROGRESS
{
"snapshot" : "2015.03.11.06.04.39",
"indices" : [ "playground_video", "playground_admin",
"playground_profile", "playground_feed" ],
"state" : "IN_PROGRESS",
"start_time" : "2015-03-11T06:04:39.044Z",
"start_time_in_millis" : 1426053879044,
"failures" : ,
"shards" : {
"total" : 0,
"failed" : 0,
"successful" : 0
}
}

curl -XDELETE

'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'

this operation never completes. I left it running for a day..still no use..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-b%3DgkVTUrXPnGsLUq7Qrt1_PaF%3Du0hQ6kfi2F7zQDEXA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

My ES version is 1.4.2, ArchLinux
But this snapshot was taken when ES was on 1.4.0, I have recently
redeployed the ES cluster to 1.4.2 and found out that a snapshot that was
taken 2 weeks back is stuck.

As I said in my first post , I tried deleting like this curl -XDELETE
'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'
But this never completes, I am afraid I cant delete the files phisically
which will mess every thing up, I have many snapshots(hourly) taken in that
repository.

Is deleting some metadata file phisically or something solves it? I am ok
even If I am going to loose this particular snapshot.

Thanks,
Pradeep.

On Thursday, April 2, 2015 at 9:39:53 AM UTC+5:30, Mark Walkom wrote:

What version are you on?

You can try deleting the snapshot and trying again.

On 1 April 2015 at 21:53, Pradeep Reddy <pradeepreddy...@gmail.com
<javascript:>> wrote:

I am using cloud aws plugin, Snapshots are taken directly to S3.
It's a 3 node cluster.

ES config:
cloud:
aws:
access_key: xxx
secret_key: yyy
protocol: http
region: us-west-1
discovery.type: ec2
discovery.ec2.groups: elasticsearch
repositories:
s3:
bucket: "xxx"
region: "us-west-1"

When I run this

curl -XGET
'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'

Part of the output is
"snapshots" : [ {
"snapshot" : "2015.03.11.06.04.39",
"repository" : "my_s3_repository",
"state" : "ABORTED",
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 17,
"failed" : 3,
"total" : 20
},

When I run

curl localhost:9200/_snapshot/_all?pretty

it shows IN_PROGRESS
{
"snapshot" : "2015.03.11.06.04.39",
"indices" : [ "playground_video", "playground_admin",
"playground_profile", "playground_feed" ],
"state" : "IN_PROGRESS",
"start_time" : "2015-03-11T06:04:39.044Z",
"start_time_in_millis" : 1426053879044,
"failures" : ,
"shards" : {
"total" : 0,
"failed" : 0,
"successful" : 0
}
}

curl -XDELETE

'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'

this operation never completes. I left it running for a day..still no
use..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4a34a807-d596-4ca7-b90b-f709f52d624b%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ac39038c-d99e-49b8-a073-102aef5cfcbf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Any solutions to this problem??

Meanwhile, I have created a ticket @ elasticsearch repo

On Wednesday, April 1, 2015 at 4:23:39 PM UTC+5:30, Pradeep Reddy wrote:

I am using cloud aws plugin, Snapshots are taken directly to S3.
It's a 3 node cluster.

ES config:
cloud:
aws:
access_key: xxx
secret_key: yyy
protocol: http
region: us-west-1
discovery.type: ec2
discovery.ec2.groups: elasticsearch
repositories:
s3:
bucket: "xxx"
region: "us-west-1"

When I run this

curl -XGET
'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39/_status?pretty'

Part of the output is
"snapshots" : [ {
"snapshot" : "2015.03.11.06.04.39",
"repository" : "my_s3_repository",
"state" : "ABORTED",
"shards_stats" : {
"initializing" : 0,
"started" : 0,
"finalizing" : 0,
"done" : 17,
"failed" : 3,
"total" : 20
},

When I run

curl localhost:9200/_snapshot/_all?pretty

it shows IN_PROGRESS
{
"snapshot" : "2015.03.11.06.04.39",
"indices" : [ "playground_video", "playground_admin",
"playground_profile", "playground_feed" ],
"state" : "IN_PROGRESS",
"start_time" : "2015-03-11T06:04:39.044Z",
"start_time_in_millis" : 1426053879044,
"failures" : ,
"shards" : {
"total" : 0,
"failed" : 0,
"successful" : 0
}
}

curl -XDELETE

'localhost:9200/_snapshot/my_s3_repository/2015.03.11.06.04.39'

this operation never completes. I left it running for a day..still no use..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/839c1768-fe16-43b2-9f66-7cdc53021582%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Joining new nodes with 1.5.x version then terminating old nodes solved the
issue.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/468a4caf-b7ed-4b72-8d3e-11943b4a863a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.