How to get S3 repository working with AWS instance profile

Elastic Stack Elasticsearch
1

Elasticsearch version: 5.2

Plugins installed: [repository-s3]

JVM version:
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode

OS version: CentOS Linux release 7.3.1611 (Core)

Description of the problem including expected versus actual behavior:
I am using repository-s3 with aws instance profile, but got Access Denied error with http code 500.

Here are the command and output:

[root@ip-10-99-3-140 bin]# curl -XPUT '10.99.3.140:9200/_snapshot/my_s3_repository_3?pretty' -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearchbucket-14lu5ab5ncv3a",
    "region": "us-west-2"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repository_3] path  is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository_3] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-NxZaN5PAQOOZH7c528Krrg/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 99B74F2EF8BBBD0C)"
      }
    }
  },
  "status" : 500
}

The ec2 instance has the iam role and policies for accessing this bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::elasticsearchbucket-14lu5ab5ncv3a/*",
                "arn:aws:s3:::elasticsearchbucket-14lu5ab5ncv3a"
            ],
            "Effect": "Allow"
        }
    ]
}

Logs can be seen from here: https://github.com/elastic/elasticsearch/issues/23780

I want to know how to get it working.

2

Any one knows the answer?

3

Just checking. Is 10.99.3.140 the master node?

Does this profile have been applied to all nodes?

4

10.99.3.140 is the master node.
There are 2 data nodes with private ip address 10.99.2.91 and 10.99.3.228.

All the 3 nodes shares the same Instance Profile and IAM role.

5

That's strange.

The recommended permissions are:

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

They are more restrictive than the one you gave.

I know that a lot of users are also using S3 with AWS profiles like this and AFAIK no one reported such a problem.

Can you run a test in a non production env? Just start a single node, apply similar permissions and create a S3 repo?

One other thing you can do is to change the logging level and see if it tells us more things?
Do you have anything set in your elasticsearch.yml file? Can you share it (and mask any credential)?

6

Here is the master's config. Data nodes' are more or less the same.

[root@ip-10-99-3-140 ~]# grep -vP '^#|^$' /etc/elasticsearch/elasticsearch.yml
cluster.name: dev-eXX-es-cluster
node.name: ip-10-99-3-140.enzow.osaas
node.attr.rack: us-west-2c
node.master: true
node.data: false
node.ingest: true
path.data: /es_data/data
path.logs: /es_data/logs
bootstrap.memory_lock: true
network.bind_host: 10.99.3.140
network.publish_host: 52.a.b.c
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts:
  - 52.xx.xx.xx
  - 52.yy.yy.yy
  - 52.zz.zz.zz
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 1
7

Is there a tutorial setting S3 repository with AWS instance profile?

I can test a single node with similar setting to see how it goes and if there is such a tutorial would be more than helpful.

8

I switched the DEBUG log level and found some weird stuff.
log: https://gist.github.com/r0c/a64d4b3f0e516e33af68888aad0e4a56#file-gistfile1-txt

  • elasticsearch indeed got credentials from ec2 instance profile
  • HEAD / and HEAD /test.file towards the s3 bucket succeeded
  • PUT /test.file failed with response code 403

@dadoonet any ideas?

9

Where is your ec2 instance running? I guess it's in us-west-2, right?

Can you run from your machine:

aws iam list-instance-profiles
10

Finally found the reason.

One of the s3 bucket polices denies the ingress traffic without "s3:x-amz-server-side-encryption: AES256" header.

So the if a change the request to the following it works very well.

[root@ip-10-99-3-140 ~]# curl -XPUT '10.99.3.140:9200/_snapshot/my_s3_repository4?pretty' -H 'Content-Type: application/json' -d'
> {
>   "type": "s3",
>   "settings": {
>     "bucket": "elasticsearchbucket-14lu5ab5ncv3a",
>     "region": "us-west-2",
>     "server_side_encryption": true
>   }
> }'
{
  "acknowledged" : true
}
11

Thanks for closing this by sharing your solution. Wondering if it is worth adding that in documentation.

12

that's will be really healpful!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.