# How to get S3 repository working with AWS instance profile

**URL:** <https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380>\
**Category:** Elasticsearch\
**Created:** [March 28, 2017, 9:26pm UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380 "2017-03-28T21:26:21Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 28, 2017, 9:26pm UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/1 "2017-03-28T21:26:21Z")

</div>

**Elasticsearch version** : 5.2

**Plugins installed** : [repository-s3]

**JVM version** :  
java version "1.8.0\_102"  
Java(TM) SE Runtime Environment (build 1.8.0\_102-b14)  
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode

**OS version** : CentOS Linux release 7.3.1611 (Core)

**Description of the problem including expected versus actual behavior** :  
I am using repository-s3 with aws instance profile, but got Access Denied error with http code 500.

Here are the command and output:

```auto
[root@ip-10-99-3-140 bin]# curl -XPUT '10.99.3.140:9200/_snapshot/my_s3_repository_3?pretty' -H 'Content-Type: application/json' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "elasticsearchbucket-14lu5ab5ncv3a",
    "region": "us-west-2"
  }
}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "repository_verification_exception",
        "reason" : "[my_s3_repository_3] path is not accessible on master node"
      }
    ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository_3] path is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-NxZaN5PAQOOZH7c528Krrg/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 99B74F2EF8BBBD0C)"
      }
    }
  },
  "status" : 500
}

```

The ec2 instance has the iam role and policies for accessing this bucket:

```auto
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::elasticsearchbucket-14lu5ab5ncv3a/*",
                "arn:aws:s3:::elasticsearchbucket-14lu5ab5ncv3a"
            ],
            "Effect": "Allow"
        }
    ]
}

```

Logs can be seen from here: [https://github.com/elastic/elasticsearch/issues/23780](https://github.com/elastic/elasticsearch/issues/23780)

I want to know how to get it working.

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 30, 2017, 9:27pm UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/2 "2017-03-30T21:27:31Z")

</div>

Any one knows the answer?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 31, 2017, 7:47am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/3 "2017-03-31T07:47:37Z")

</div>

Just checking. Is `10.99.3.140` the master node?

Does this profile have been applied to all nodes?

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 31, 2017, 9:53am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/4 "2017-03-31T09:53:09Z")

</div>

10.99.3.140 is the master node.  
There are 2 data nodes with private ip address 10.99.2.91 and 10.99.3.228.

All the 3 nodes shares the same Instance Profile and IAM role.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 31, 2017, 10:27am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/5 "2017-03-31T10:27:59Z")

</div>

That's strange.

The recommended permissions are:

```auto
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::snaps.example.com/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

```

They are more restrictive than the one you gave.

I know that a lot of users are also using S3 with AWS profiles like this and AFAIK no one reported such a problem.

Can you run a test in a non production env? Just start a single node, apply similar permissions and create a S3 repo?

One other thing you can do is to change the logging level and see if it tells us more things?  
Do you have anything set in your `elasticsearch.yml` file? Can you share it (and mask any credential)?

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 31, 2017, 10:45am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/6 "2017-03-31T10:45:10Z")

</div>

Here is the master's config. Data nodes' are more or less the same.

```auto
[root@ip-10-99-3-140 ~]# grep -vP '^#|^$' /etc/elasticsearch/elasticsearch.yml
cluster.name: dev-eXX-es-cluster
node.name: ip-10-99-3-140.enzow.osaas
node.attr.rack: us-west-2c
node.master: true
node.data: false
node.ingest: true
path.data: /es_data/data
path.logs: /es_data/logs
bootstrap.memory_lock: true
network.bind_host: 10.99.3.140
network.publish_host: 52.a.b.c
http.port: 9200
transport.tcp.port: 9300
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts:
  - 52.xx.xx.xx
  - 52.yy.yy.yy
  - 52.zz.zz.zz
discovery.zen.minimum_master_nodes: 1
node.max_local_storage_nodes: 1

```

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 31, 2017, 10:48am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/7 "2017-03-31T10:48:10Z")

</div>

Is there a tutorial setting S3 repository with AWS instance profile?

I can test a single node with similar setting to see how it goes and if there is such a tutorial would be more than helpful.

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [March 31, 2017, 11:21am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/8 "2017-03-31T11:21:21Z")

</div>

I switched the DEBUG log level and found some weird stuff.  
log: [https://gist.github.com/r0c/a64d4b3f0e516e33af68888aad0e4a56#file-gistfile1-txt](https://gist.github.com/r0c/a64d4b3f0e516e33af68888aad0e4a56#file-gistfile1-txt)

- elasticsearch indeed got credentials from ec2 instance profile
- HEAD / and HEAD /test.file towards the s3 bucket succeeded
- PUT /test.file failed with response code 403

@dadoonet any ideas?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [March 31, 2017, 12:28pm UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/9 "2017-03-31T12:28:09Z")

</div>

Where is your ec2 instance running? I guess it's in `us-west-2`, right?

Can you run from your machine:

```auto
aws iam list-instance-profiles

```

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [April 1, 2017, 10:55am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/10 "2017-04-01T10:55:00Z")

</div>

Finally found the reason.

One of the s3 bucket polices **denies** the ingress traffic without "s3:x-amz-server-side-encryption: AES256" header.

So the if a change the request to the following it works very well.

```auto
[root@ip-10-99-3-140 ~]# curl -XPUT '10.99.3.140:9200/_snapshot/my_s3_repository4?pretty' -H 'Content-Type: application/json' -d'
> {
> "type": "s3",
> "settings": {
> "bucket": "elasticsearchbucket-14lu5ab5ncv3a",
> "region": "us-west-2",
> "server_side_encryption": true
> }
> }'
{
  "acknowledged" : true
}

```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [April 1, 2017, 11:38am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/11 "2017-04-01T11:38:42Z")

</div>

Thanks for closing this by sharing your solution. Wondering if it is worth adding that in documentation.

---

<div class="post-metadata">

**Author:** ![r0c](https://avatars.discourse-cdn.com/v4/letter/r/4bbf92/32.png) [@r0c](https://discuss.elastic.co/u/r0c)\
**Post date:** [April 1, 2017, 11:53am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/12 "2017-04-01T11:53:25Z")

</div>

that's will be really healpful!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2017, 11:53am UTC](https://discuss.elastic.co/t/how-to-get-s3-repository-working-with-aws-instance-profile/80380/13 "2017-04-29T11:53:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.