# Registering S3 repository to ElasticSearch running on AWS ECS

**URL:** <https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556>\
**Category:** Elasticsearch\
**Created:** [September 27, 2016, 5:05am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556 "2016-09-27T05:05:25Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 27, 2016, 5:05am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/1 "2016-09-27T05:05:25Z")

</div>

HI.

I'm trying to register S3 repository to ES(v.2.4) which is docker container running on AWS ECS cluster instance.

config/elasticsearch.yml

```
 repositories:
      s3:
        bucket: "my.bucket"
        region: "ap-northeast-1"

```

S3 bucket policy

```
   {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:ListBucket",
    				"s3:GetBucketLocation",
    				"s3:ListBucketMultipartUploads",
    				"s3:ListBucketVersions"
    			],
    			"Resource": "arn:aws:s3:::my.bucket"
    		},
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:GetObject",
    				"s3:PutObject",
    				"s3:DeleteObject",
    				"s3:AbortMultipartUpload",
    				"s3:ListMultipartUploadParts"
    			],
    			"Resource": "arn:aws:s3:::my.bucket/*"
    		}
    	]
    }

```

I gave S3FullAccess to ECS task roll.

```
GET _nodes/?pretty
    ...
    "repositories": {
              "s3": {
                "bucket": "my.bucket",
                "region": "ap-northeast-1"
              }
    ...

GET _snapshot/?pretty
{}

```

I got an error like this, when I tried to register repository.

```
PUT _snapshot/s3_snapshot_repository
    {
      "type": "s3",
      "settings": {
        "bucket": "my.bucket",
        "region": "ap-northeast-1"
      }
    }
{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[s3_snapshot_repository] path is not accessible on master node"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[s3_snapshot_repository] path is not accessible on master node",
      "caused_by": {
         "type": "i_o_exception",
         "reason": "Unable to upload object tests-Xa3F_VHATDuJ45kz5jo6rg/master.dat-temp",
         "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied"
         }
      }
   },
   "status": 500
}

```

However, test files were generated to S3 bucket.

 ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c6cbaf26491e42245afb65c1a89f2b6458ef9dbf.png)

What should I do to solve this error?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 5:52am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/2 "2016-09-27T05:52:33Z")

</div>

Which version?

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 27, 2016, 5:55am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/3 "2016-09-27T05:55:09Z")

</div>

Sorry, I forgot to mention about it.  
I'm using version 2.4.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 6:30am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/4 "2016-09-27T06:30:16Z")

</div>

Thanks! I had another similar report on 2.4 so I'll try to reproduce and come back later.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 7:29am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/5 "2016-09-27T07:29:25Z")

</div>

Can you try a similar test without docker? Like: start an EC2 instance and run something like:

```auto
wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz
tar xzf elasticsearch-2.4.0.tar.gz 
cd elasticsearch-2.4.0
bin/plugin install cloud-aws
vi config/elasticsearch.yml 

```

Then add your cloud settings:

```auto
cloud.aws.access_key: "KEY"
cloud.aws.secret_key: "SECRET"
cloud.aws.region: REGION

```

Then launch:

```auto
bin/elasticsearch

```

In another terminal:

```auto
curl -XDELETE 127.0.0.1:9200/_snapshot/my_s3_repository?pretty
curl -XPUT '127.0.0.1:9200/_snapshot/my_s3_repository?pretty' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "your-bucket-here",
    "region": "your-region-here"
  }
}'
curl -XDELETE 127.0.0.1:9200/foo?pretty
curl -XPUT 127.0.0.1:9200/foo/doc/1?pretty -d '{ "foo": "bar" }'
curl -XPUT "127.0.0.1:9200/_snapshot/my_s3_repository/snapshot_1?wait_for_completion=true&pretty"

```

This is working for me on a one node cluster with no docker layer involved.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 7:42am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/6 "2016-09-27T07:42:17Z")

</div>

For the record, I tried with a 2 nodes cluster and it went well.

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 27, 2016, 9:29am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/7 "2016-09-27T09:29:08Z")

</div>

Thanks for replying.

I tried as you mentioned, but still got same error.

```
curl -X PUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -d '{
  "type": "s3",
    "settings": {
      "bucket": "my.bucket",
      "region": "ap-northeast-1"
    }
}'

{
  "error" : {
    "root_cause" : [ {
      "type" : "repository_verification_exception",
      "reason" : "[my_s3_repository] path is not accessible on master node"
    } ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] path is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-R455L68wQiSdCB0hhjuUaQ/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D7A94303C3A9214)"
      }
    }
  },
  "status" : 500
}
```

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 10:07am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/8 "2016-09-27T10:07:23Z")

</div>

Would it be possible to try with this?

```auto
{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

```

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 27, 2016, 10:24am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/9 "2016-09-27T10:24:43Z")

</div>

Should I state it as IAM policy, not Bucket policy?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 27, 2016, 10:39am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/10 "2016-09-27T10:39:55Z")

</div>

Yes. Read [https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-repository.html#cloud-aws-repository-permissions](https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-repository.html#cloud-aws-repository-permissions)

> This may be configured through the AWS IAM console, by creating a Custom Policy, and using a Policy Document similar to this (changing [snaps.example.com](http://snaps.example.com) to your bucket name).

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 28, 2016, 4:30am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/11 "2016-09-28T04:30:57Z")

</div>

Thanks.  
I have already given s3 full access to the ecs task roll.

However, after adding everyone to bucket policy acl, all processes went well.  
And according to IAM history, the S3FullAccess policy was not used.

So I guessed that the aws-cloud-plugin is not using sdk, is it right ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 28, 2016, 5:21am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/12 "2016-09-28T05:21:26Z")

</div>

It is using AWS Java SDK.

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 28, 2016, 5:46am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/13 "2016-09-28T05:46:56Z")

</div>

I thought AWS cloud plugin needs only IAM role, which is assigned to the instance that ElasticSearch is running on, to communicate with S3.  
So, I just gave S3FullAccess to ECS task role and permitted the role in S3 bucket policy.

But actually, AWS cloud plugin needs user credential keys.  
I must give the credentials to the plugin, and the credentials must be assigned to the user who has S3 access authority.

These my understandings are correct?  
Sorry for my odd English.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 28, 2016, 10:55am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/14 "2016-09-28T10:55:31Z")

</div>

> [@111131](#):
>
> But actually, AWS cloud plugin needs user credential keys.

I don't think you **must** set `key/secret`. If you are running on AWS, using a policy IAM thing should be ok.  
This is how we are initializing the client: [https://github.com/elastic/elasticsearch/blob/2.4/plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java#L135-L145](https://github.com/elastic/elasticsearch/blob/2.4/plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java#L135-L145)

```
    if (account == null && key == null) {
        credentials = new AWSCredentialsProviderChain(
                new EnvironmentVariableCredentialsProvider(),
                new SystemPropertiesCredentialsProvider(),
                new InstanceProfileCredentialsProvider()
        );
    } else {
        credentials = new AWSCredentialsProviderChain(
                new StaticCredentialsProvider(new BasicAWSCredentials(account, key))
        );
    }

```

So you have multiple options available here.

> [@111131](#):
>
> Sorry for my odd English.

Perfectly fine and clear. English is not my native language either. 🙂

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 29, 2016, 8:53am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/15 "2016-09-29T08:53:37Z")

</div>

Thanks,

I understand aws-cloud-plugin doesn't necessarily need a credential key set.

But, I think aws-cloud-plugin is not using ECS task role in my case.

I confirmed ECS task role was enabled by below process.

[http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)  
In docker container which ElasticSearch running on, the command "_curl 169.254.170.2_$AWS\_CONTAINER\_CREDENTIALS\_RELATIVE\_URI" returns ECS task role.

[http://docs.aws.amazon.com/cli/latest/reference/sts/get-caller-identity.html](http://docs.aws.amazon.com/cli/latest/reference/sts/get-caller-identity.html)  
I also installed aws-sdk to same container and executed get-caller-identity.  
It returned ECS task role.

So, It can be said ECS task role was enabled.

As I wrote firstly, when I gave S3FullAccess to ECS task role, the snapshot process failed.  
But when I gave same authority to IAM role which is belongs to ECS instance, the process went well.  
Bucket policy permits the access from account root, so I think aws-cloud-plugin is not using ECS task role, but IAM role of ECS instance.

In your quote, maybe, the InstanceProfileCredentialsProvider handles IAM role and I feel this class doesn't care ECS task role from the name. (Sorry for not reading actual codes.)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 30, 2016, 10:17am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/16 "2016-09-30T10:17:02Z")

</div>

Ok. I missed actually that you were running with ECS. I thought you were using EC2.  
I have absolutely no experience with ECS and I'm unsure if we can "fix" anything on our end.

May be updating the AWS SDK to a more recent version could help cloud-aws plugin to use ECS task role but again I have no clue here.

---

<div class="post-metadata">

**Author:** ![111131](https://avatars.discourse-cdn.com/v4/letter/1/f07891/32.png) [@111131](https://discuss.elastic.co/u/111131)\
**Post date:** [September 30, 2016, 10:21am UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/17 "2016-09-30T10:21:01Z")

</div>

OK. Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:15pm UTC](https://discuss.elastic.co/t/registering-s3-repository-to-elasticsearch-running-on-aws-ecs/61556/18 "2017-07-05T22:15:51Z")

</div>