Recently I've been seeing warnings of 'failed to read commit point [snapshotWhichIDeletedAWhileAgo]' when trying to delete snapshots. E.g.
2016-01-29T20:52:36.805+00:00","Deleting snapshot foo-1"
2016-01-29T21:05:12.714+00:00","Finished deleting snapshot foo-1"
2016-01-29T21:05:12.715+00:00","Deleting snapshot foo-2"
2016-01-29T21:13:15.225+00:00","WARN [h[mynode][snapshot][T#191]] [index.snapshots.blobstore ]: [mynode] failed to read commit point [snapshot-foo-1]]
java.io.FileNotFoundException: Blob object [snapshot-foo-1] not found: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: XYZ)
at org.elasticsearch.cloud.aws.blobstore.S3BlobContainer.openInput(S3BlobContainer.java:90)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.buildBlobStoreIndexShardSnapshots(BlobStoreIndexShardRepository.java:401)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository$Context.delete(BlobStoreIndexShardRepository.java:299)
at org.elasticsearch.index.snapshots.blobstore.BlobStoreIndexShardRepository.delete(BlobStoreIndexShardRepository.java:218)
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.deleteSnapshot(BlobStoreRepository.java:315)
at org.elasticsearch.snapshots.SnapshotsService$10.run(SnapshotsService.java:1317)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
I am doing the deletion via
es.admin().cluster().deleteSnapshot(new DeleteSnapshotRequest(repository, "foo-1")).actionGet();
I recently upgraded to 1.7.3; I'm not positive if the error started with the upgrade or after it. As far as I can tell looking at the source there's no way the call in the stacktrace should be being made with snapshot id 'foo-1' after deletion of that snapshot has completed. Any ideas?