I’m having trouble creating a S3 snapshot repository for ES 6.2 on DigitalOcean. While the S3 repository plugin works fine with ES 5.6, I can’t get it to work on ES 6.2.
It appears you are running your own s3 clone. Note that we do not explicitly support this. Have you checked logs in your s3 server to see if an error exists on that side?
Also note that region was removed as parameter (although I might be mixed up on the timeline when it was removed, but I thought it was for 6.0). You should be getting an unknown parameter error for that...
Additionally, by specifying your access and secret key inside the keystore, you should not need either the allow_insecure_settings sysprop or the access/secret key settings in your request.
Yes I'm using DigitalOcean Spaces not AWS, but work's fine on ES 5.6 and can't really see why it wouldn’t on 6.2?
Had a look at this https://www.elastic.co/guide/en/elasticsearch/reference/6.2/breaking_60_plugins_changes.html and can’t really see, if there is anything I should do different from what I’m doing on ES 5.6. Have tried both with allow_insecure_settings=true and access_key/secret_key in the request as well as without allow_insecure_settings and access_key/secret_key in the keystore – but gives me the same error response.
Unfortunately I don’t know, if I can access the log in DigitalOcean.
[2018-03-21T13:05:16,263][INFO ][o.e.r.RepositoriesService] [MediaData_ESmain] update repository [snapshot]
[2018-03-21T13:05:16,409][WARN ][r.suppressed ] path: /_snapshot/snapshot, params: {pretty=, repository=snapshot}
org.elasticsearch.common.blobstore.BlobStoreException: Failed to check if blob [master.dat-temp] exists
at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:78) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:99) ~[?:?]
at org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:574) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.repositories.RepositoriesService.verifyRepository(RepositoriesService.java:211) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:414) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.repositories.RepositoriesService$VerifyingRegisterRepositoryListener.onResponse(RepositoriesService.java:399) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.AckedClusterStateUpdateTask.onAllNodesAcked(AckedClusterStateUpdateTask.java:64) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.MasterService$SafeAckedClusterStateTaskListener.onAllNodesAcked(MasterService.java:526) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.MasterService$AckCountDownListener.onNodeAck(MasterService.java:626) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.MasterService$DelegetingAckListener.onNodeAck(MasterService.java:566) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery$1.onNewClusterStateProcessed(ZenDiscovery.java:362) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.PendingClusterStatesQueue.markAsProcessed(PendingClusterStatesQueue.java:177) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.discovery.zen.ZenDiscovery$3.clusterStateProcessed(ZenDiscovery.java:802) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.ClusterApplierService$SafeClusterStateTaskListener.clusterStateProcessed(ClusterApplierService.java:540) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:491) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.2.3.jar:6.2.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 34D7FE81D3A09BCD; S3 Extended Request ID: nrF6AslBHKrRlT4A04YgkBQhFl0wtoTebFNbOr8dHbqcpdP+wyBGcP689THQgckMGKp5SjbhGH8=)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1253) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1228) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1304) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.lambda$blobExists$0(S3BlobContainer.java:76) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_151]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivileged(SocketAccess.java:42) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobContainer.blobExists(S3BlobContainer.java:76) ~[?:?]
... 22 more
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.