S3 permissions

[thoth]

The documentation has become outdated on the S3 permissions.

https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-s3-repository.html#repository-s3-permissions

If you try to apply this policy AWS complains about missing a principal. So I tried the policy generator, but I am unable to come up with a magical combination of something that works.

The credentials were created using the keystore tool:

elasticsearch-keystore add s3.client.default.access_key
elasticsearch-keystore add s3.client.default.secret_key

I consistently get the following:

    {
      "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[AWSserviceaccount] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[AWSserviceaccount] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Unable to upload object [tests-xxxxxxxxxxxxxx/master.dat] using a single upload",
      "caused_by": {
        "type": "sdk_client_exception",
        "reason": "Unable to load credentials from service endpoint",
        "caused_by": {
          "type": "socket_timeout_exception",
          "reason": "Connect timed out"
        }
      }
    }
      },
      "status": 500
    }

Can we get an update to the docs that gives the correct method of creating the IAM user + bucket?

[spinscale (Alexander Reelsen)]

Hey,

can you share the error message of the missing principal? The exception you pasted looks like a network issue, as the network connection seemed to have timed out...

--Alex

[thoth]

yep, when saving the given json (*with the ARN from my bucket replaced in there) from the link above, I get:

Error

Missing required field Principal

So instead I generated a config like:

{
    "Version": "2012-10-17",
    "Id": "Policy888bignum888",
    "Statement": [
        {
            "Sid": "888bigsid888",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::888bignum888:user/myserviceaccount"
            },
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::myarn"
        }
    ]
}

but this is not working. Translating the changes back into the documented policy I get something like:

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::876bignum876:user/myserviceaccount"
      },
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Principal": {
          "AWS": "arn:aws:iam::876bignum876:user/myserviceaccount"
      },
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

but again not working as well

[thoth]

@spinscale to test the network connection I executed a shell inside my elasticsearch-master-0 host with:

kubectl exec -it elasticsearch-master-0 -- /bin/bash

and then tried a verbose curl on the main endpoint:

curl s3.amazonaws.com -v
* About to connect() to s3.amazonaws.com port 80 (#0)
*   Trying 52.216.204.205...
* Connected to s3.amazonaws.com (52.216.204.205) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: s3.amazonaws.com
> Accept: */*
> 
< HTTP/1.1 307 Temporary Redirect
< x-amz-id-2: dqTEXiH0LTQr4OyT5edurqjPbZLvdnCWHVyfA0pfX9ObfzbskQuA9Y7wY7VgESIHN95Swmlu0Yc=
< x-amz-request-id: 066D2981B33F5D86
< Date: Tue, 24 Mar 2020 13:41:28 GMT
< Location: https://aws.amazon.com/s3/
< Content-Length: 0
< Server: AmazonS3
< 
* Connection #0 to host s3.amazonaws.com left intact

Which seems like a good connection to me. I followed through to curl https://aws.amazon.com/s3/ -v and I get the same results that I do on my local laptop.