Can't create s3 repository for backup in AWS Beijing region

k-troska · July 25, 2018, 1:27pm

Elasticsearch version (bin/elasticsearch --version):
Version: 6.3.2, Build: oss/deb/053779d/2018-07-20T05:20:23.451332Z, JVM: 1.8.0_171

Plugins installed:
ingest-geoip
ingest-user-agent
repository-s3

JVM version (java -version):
openjdk version "1.8.0_171"
OpenJDK Runtime Environment (build 1.8.0_171-8u171-b11-1~bpo8+1-b11)
OpenJDK 64-Bit Server VM (build 25.171-b11, mixed mode)

OS version (uname -a if on a Unix-like system):
Linux [name redacted] 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u3 (2017-08-15) x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
Can't add repository-s3 in cn-north-1 region (Beijing), with following error:
Request
curl -s -X PUT "http://[hostname]:9200/_snapshot/s3_repository?pretty" -d @repo.json -H "Content-type:application/json"
repo.json
{ "type": "s3", "settings": { "bucket": "[name redacted]" } }
Output

{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_exception",
        "reason" : "[s3_repository] failed to create repository"
      }
    ],
    "type" : "repository_exception",
    "reason" : "[s3_repository] failed to create repository",
    "caused_by" : {
      "type" : "amazon_s3_exception",
      "reason" : "Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: C171506AC1F99990; S3 Extended Request ID: K8qtQddMD7JX+Y/B58oz04rE/HlPeHlxQyuK0e3HLSHk0e3HhzWSL2sBC+OJ+tj2BXH4EXfFsP4=)"
    }
  },
  "status" : 500
}

Steps to reproduce:

Please include a minimal but complete recreation of the problem, including
(e.g.) index creation, mappings, settings, query etc. The easier you make for
us to reproduce it, the more likely that somebody will take the time to look at it.

install elasticsearch-oss
install repository-s3 plugin
try to add repository

Provide logs (if relevant):

Stack trace:

[2018-07-25T19:39:02,255][WARN ][r.suppressed             ] path: /_snapshot/s3_repository, params: {pretty=, repository=s3_repository}
org.elasticsearch.repositories.RepositoryException: [s3_repository] failed to create repository
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:388) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService.registerRepository(RepositoriesService.java:356) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService.access$100(RepositoriesService.java:55) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.repositories.RepositoriesService$1.execute(RepositoriesService.java:108) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:45) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:630) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:267) ~[elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:197) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:132) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:626) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [elasticsearch-6.3.2.jar:6.3.2]
	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [elasticsearch-6.3.2.jar:6.3.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request ID: C76856AFD42A7C18; S3 Extended Request ID: Dwj3TJ9dSlw6KYB4Xq9SZEbpntDIx+4upQcq2krrl2dktf2flSJV0u2Kbpm4QGy0XfxhKOCj9tM=)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1639) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1304) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649) ~[?:?]
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4247) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4194) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.headBucket(AmazonS3Client.java:1326) ~[?:?]
	at com.amazonaws.services.s3.AmazonS3Client.doesBucketExist(AmazonS3Client.java:1266) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.lambda$new$0(S3BlobStore.java:72) ~[?:?]
	at org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:57) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_171]
	at org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:56) ~[?:?]
	at org.elasticsearch.repositories.s3.S3BlobStore.<init>(S3BlobStore.java:71) ~[?:?]
	at org.elasticsearch.repositories.s3.S3Repository.<init>(S3Repository.java:183) ~[?:?]
	at org.elasticsearch.repositories.s3.S3RepositoryPlugin.lambda$getRepositories$1(S3RepositoryPlugin.java:76) ~[?:?]
	at org.elasticsearch.repositories.RepositoriesService.createRepository(RepositoriesService.java:383) ~[elasticsearch-6.3.2.jar:6.3.2]
	... 16 more

k-troska · July 25, 2018, 1:30pm

Generated with help of: https://github.com/elastic/ansible-elasticsearch
cat /etc/elasticsearch/es01/elasticsearch.yml


bootstrap.memory_lock: true
cluster.name: APAC-logs
cluster.routing.allocation.awareness.attributes: zone
cluster.routing.allocation.awareness.force.zone.values: cn-north-1a,cn-north-1b
discovery.zen.ping.unicast.hosts: es01.[redacted],es02.[redacted]
network.host: 10.16.40.182
node.attr.zone: cn-north-1a
node.data: true
node.master: true
node.name: es01



#################################### Paths ####################################

# Path to directory containing configuration (this file and logging.yml):


path.data: /var/lib/elasticsearch/es01.elk.[redacted]-es01

path.logs: /var/log/elasticsearch/es01.elk.[redacted]-es01


action.auto_create_index: true

For permission I am using Iam policy/role:

{
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListBucketVersions"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws-cn:s3:::[redacted]"
            ]
        },
        {
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws-cn:s3:::[redacted]/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

dadoonet · July 25, 2018, 1:55pm

I already linked to https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-client.html in the github issue you opened.

Set endpoint.

k-troska · July 25, 2018, 2:50pm

Thank you it helped, seams to work after I added
s3.client.default.endpoint: s3.cn-north-1.amazonaws.com.cn
to elasticsearch.yml on both nodes and restarted them.

dadoonet · July 25, 2018, 5:03pm

I'm curious about the bucket name you used then?
Any chance you can DM it to me if you don't want to share it publicly?

dadoonet · July 31, 2018, 12:57pm

For the record I opened this PR to fix the documentation:

system · August 28, 2018, 12:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.