S3 repository Snapshot fails on ES 6.2

emillykkejensen · March 16, 2018, 1:21pm

I’m having trouble creating a S3 snapshot repository for ES 6.2 on DigitalOcean. While the S3 repository plugin works fine with ES 5.6, I can’t get it to work on ES 6.2.

Here is what I have done:
1: Installed ES 6.2 with the S3 snapshot repository and reindexed some data over to the new server.
2: Completed the steps here: https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/repository-s3-client.html and created AWS access keys in the elasticsearch keystore.
3: Realized that there seems to be a problem with that (https://github.com/elastic/elasticsearch/issues/27765 and Failed to create S3 repository with ES 6.0) and therefor added this line: -Des.allow_insecure_settings=true to the file jvm.options in /etc/elasticsearch
4: Still getting an error, every time I try to create the repository

The code I try to run:

curl -XPUT "http://localhost:9200/_snapshot/snapshot_ESmain" -H "Content-Type: application/json" -d {
    "type":"s3",
    "settings":{
         "bucket":"mediadata",
        "region":"AMS3",
         "endpoint":"ams3.digitaloceanspaces.com",
         "protocol":"https",
         "base_path":"snapshot_ESmain",
         "compress":true,
         "access_key":"MYNOTSOSECRETKET",
         "secret_key":"MYSECRETKEY"}
    }

…and the response I get:

{"error":{
     "root_cause":[{
          "type":"blob_store_exception",
          "reason":"Failed to check if blob [master.dat-temp] exists"}],
     "type":"blob_store_exception",
     "reason":"Failed to check if blob [master.dat-temp] exists",
     "caused_by":{
          "type":"amazon_s3_exception",
          "reason":"Forbidden (
               Service: Amazon S3;
               Status Code: 403;
               Error Code: 403 Forbidden;
               Request ID: THEREQUESTID;
               S3 Extended Request ID: THEEXTENDEDREQUESTID)"}},
"status":500}

What am I doing wrong?

rjernst · March 16, 2018, 5:34pm

It appears you are running your own s3 clone. Note that we do not explicitly support this. Have you checked logs in your s3 server to see if an error exists on that side?

Also note that region was removed as parameter (although I might be mixed up on the timeline when it was removed, but I thought it was for 6.0). You should be getting an unknown parameter error for that...

Additionally, by specifying your access and secret key inside the keystore, you should not need either the allow_insecure_settings sysprop or the access/secret key settings in your request.

emillykkejensen · March 18, 2018, 4:54pm

Yes I'm using DigitalOcean Spaces not AWS, but work's fine on ES 5.6 and can't really see why it wouldn’t on 6.2?

Had a look at this https://www.elastic.co/guide/en/elasticsearch/reference/6.2/breaking_60_plugins_changes.html and can’t really see, if there is anything I should do different from what I’m doing on ES 5.6. Have tried both with allow_insecure_settings=true and access_key/secret_key in the request as well as without allow_insecure_settings and access_key/secret_key in the keystore – but gives me the same error response.

Unfortunately I don’t know, if I can access the log in DigitalOcean.

emillykkejensen · March 21, 2018, 1:07pm

Is this of any use?? It's from the log:

[2018-03-21T13:05:16,263][INFO ][o.e.r.RepositoriesService] [MediaData_ESmain] update repository [snapshot]
[2018-03-21T13:05:16,409][WARN ][r.suppressed             ] path: /_snapshot/snapshot, params: {pretty=, repository=snapshot}
org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [master.dat-temp] exists
	at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:78) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:99) ~[?:?]
	at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:574) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:211) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:414) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:399) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:64) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:526) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:626) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.MasterService$DelegetingAckListener.onNodeAck(MasterService.java:566) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.discovery.zen.ZenDiscovery$1.onNewClusterStateProcessed(ZenDiscovery.java:362) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.discovery.zen.PendingClusterStatesQueue.markAsProcessed(PendingClusterStatesQueue.java:177) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.discovery.zen.ZenDiscovery$3.clusterStateProcessed(ZenDiscovery.java:802) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.ClusterApplierService$SafeClusterStateTaskListener.clusterStateProcessed(ClusterApplierService.java:540) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:491) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.3.jar:6.2.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 34D7FE81D3A09BCD; S3 Extended Request ID: nrF6AslBHKrRlT4A04YgkBQhFl0wtoTebFNbOr8dHbqcpdP+wyBGcP689THQgckMGKp5SjbhGH8=)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1304) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:76) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_151]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:76) ~[?:?]
	... 22 more

system · April 18, 2018, 1:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.