S3 gateway not recovering after restart

Elastic Stack Elasticsearch
1

Hi,

I am currently trying out the s3 gateway and upon a cluster restart I
am getting the following error message:

[2011-05-16 09:58:48,052][WARN ][index.gateway.s3 ] [Burner]
[testindex][0] listed commit_point [commit-24m]/[2758], but not all
files exists, ignoring
[2011-05-16 09:58:48,052][WARN ][indices.cluster ] [Burner]
[testindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[testindex][0] No commit point data is available in gateway
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:
423)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:144)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I though the s3 storage would have been sufficient in order to ensure
consistent indices. As I am currently using no replicas, is there an
alternative to just deleting
the indices in order to get the cluster to green again? What would be
a good strategy to avoid this kind of problems in the future?

Thanks,
Michel

2

It seems like its missing some fields from s3 to do proper recovery. The logic is that files are only deleted once a new snapshot has been properly executed.

Back to your question, do you have a single node or a cluster of nodes? Running with no replicas and a shared gateway will mean failover will be slow since all the data will need to be "downloaded" from the shared gateway.
On Monday, May 16, 2011 at 11:50 AM, Michel Conrad wrote:

Hi,

I am currently trying out the s3 gateway and upon a cluster restart I
am getting the following error message:

[2011-05-16 09:58:48,052][WARN ][index.gateway.s3 ] [Burner]
[testindex][0] listed commit_point [commit-24m]/[2758], but not all
files exists, ignoring
[2011-05-16 09:58:48,052][WARN ][indices.cluster ] [Burner]
[testindex][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[testindex][0] No commit point data is available in gateway
at
org.elasticsearch.index.gateway.blobstore.BlobStoreIndexShardGateway.recover(BlobStoreIndexShardGateway.java:
423)
at org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:144)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

I though the s3 storage would have been sufficient in order to ensure
consistent indices. As I am currently using no replicas, is there an
alternative to just deleting
the indices in order to get the cluster to green again? What would be
a good strategy to avoid this kind of problems in the future?

Thanks,
Michel