Environment info:
Server: Ubuntu 16.04.4
Elasticsearch: 6.2.4
I'm trying to setup a new repository in an S3 bucket hosted in GovCloud. When I run: curl -XPUT 'http://localhost:9200/_snapshot/s3_repository?verfiy=false&pretty' -H 'Content-Type: application/json' -d' { "type": "s3","settings": { "bucket": "s3_bucket_name", "endpoint": "s3-us-gov-west-1.amazonaws.com" }}' I get the error below returned.
`{
"error" : {
"root_cause" : [
{
"type" : "amazon_s3_exception",
"reason" : "amazon_s3_exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: XXXXXXXXXX; S3 Extended Request ID: XXXXXXXXXXXX"
}
],
"type" : "repository_exception",
"reason" : "[s3_repository] failed to create repository",
"caused_by" : {
"type" : "amazon_s3_exception",
"reason" : "amazon_s3_exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: XXXXXXXXXX; S3 Extended Request ID: XXXXXXXXXXXX"
}
},
"status" : 500
}`
The IAM role is set to allow all permissions and I've set s3.client.default.access_key and s3.client.default.secret_key explicitly with named user keys in the keystore. I'm at a loss to what I'm missing. In a standard AWS account this exact same command is working on the same setup (same access between EC2 and S3, roles, etc.). The relevant portion of the elasticsearch log is below.
`Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: amazon_s3_exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: XXXXXXXXXXXXX; S3 Extended Request ID: XXXXXXXXXXXXXXXX
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]
at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:384) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:357) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:56) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:109) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:643) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:273) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:198) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:133) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:573) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_162]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_162]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_162]
`