Failed to read commit point snapshot error


#1

Recently I've been seeing warnings of 'failed to read commit point [snapshotWhichIDeletedAWhileAgo]' when trying to delete snapshots. E.g.

2016-01-29T20:52:36.805+00:00","Deleting snapshot foo-1"
2016-01-29T21:05:12.714+00:00","Finished deleting snapshot foo-1"
2016-01-29T21:05:12.715+00:00","Deleting snapshot foo-2"
2016-01-29T21:13:15.225+00:00","WARN [h[mynode][snapshot][T#191]] [index.snapshots.blobstore ]: [mynode] failed to read commit point [snapshot-foo-1]]
java.io.FileNotFoundException: Blob object [snapshot-foo-1] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: XYZ)
	at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.openInput(S3BlobContainer.java:90)
	at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.buildBlobStoreIndexShardSnapshots(BlobStoreIndexShardRepository.java:401)
	at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.delete(BlobStoreIndexShardRepository.java:299)
	at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.delete(BlobStoreIndexShardRepository.java:218)
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.deleteSnapshot(BlobStoreRepository.java:315)
	at org.elasticsearch.snapshots.SnapshotsService$10.run(SnapshotsService.java:1317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I am doing the deletion via
es.admin().cluster().deleteSnapshot(new DeleteSnapshotRequest(repository, "foo-1")).actionGet();

I recently upgraded to 1.7.3; I'm not positive if the error started with the upgrade or after it. As far as I can tell looking at the source there's no way the call in the stacktrace should be being made with snapshot id 'foo-1' after deletion of that snapshot has completed. Any ideas?


(system) #2