Registering S3 repository to ElasticSearch running on AWS ECS

HI.

I'm trying to register S3 repository to ES(v.2.4) which is docker container running on AWS ECS cluster instance.

config/elasticsearch.yml

 repositories:
      s3:
        bucket: "my.bucket"
        region: "ap-northeast-1"

S3 bucket policy

   {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:ListBucket",
    				"s3:GetBucketLocation",
    				"s3:ListBucketMultipartUploads",
    				"s3:ListBucketVersions"
    			],
    			"Resource": "arn:aws:s3:::my.bucket"
    		},
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:GetObject",
    				"s3:PutObject",
    				"s3:DeleteObject",
    				"s3:AbortMultipartUpload",
    				"s3:ListMultipartUploadParts"
    			],
    			"Resource": "arn:aws:s3:::my.bucket/*"
    		}
    	]
    }

I gave S3FullAccess to ECS task roll.

GET _nodes/?pretty
    ...
    "repositories": {
              "s3": {
                "bucket": "my.bucket",
                "region": "ap-northeast-1"
              }
    ...

GET _snapshot/?pretty
{}

I got an error like this, when I tried to register repository.

PUT _snapshot/s3_snapshot_repository
    {
      "type": "s3",
      "settings": {
        "bucket": "my.bucket",
        "region": "ap-northeast-1"
      }
    }
{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[s3_snapshot_repository] path  is not accessible on master node"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[s3_snapshot_repository] path  is not accessible on master node",
      "caused_by": {
         "type": "i_o_exception",
         "reason": "Unable to upload object tests-Xa3F_VHATDuJ45kz5jo6rg/master.dat-temp",
         "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied"
         }
      }
   },
   "status": 500
}

However, test files were generated to S3 bucket.

What should I do to solve this error?

Which version?

Sorry, I forgot to mention about it.
I'm using version 2.4.

Thanks! I had another similar report on 2.4 so I'll try to reproduce and come back later.

Can you try a similar test without docker? Like: start an EC2 instance and run something like:

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz
tar xzf elasticsearch-2.4.0.tar.gz 
cd elasticsearch-2.4.0
bin/plugin install cloud-aws
vi config/elasticsearch.yml 

Then add your cloud settings:

cloud.aws.access_key: "KEY"
cloud.aws.secret_key: "SECRET"
cloud.aws.region: REGION

Then launch:

bin/elasticsearch

In another terminal:

curl -XDELETE 127.0.0.1:9200/_snapshot/my_s3_repository?pretty
curl -XPUT '127.0.0.1:9200/_snapshot/my_s3_repository?pretty' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "your-bucket-here",
    "region": "your-region-here"
  }
}'
curl -XDELETE 127.0.0.1:9200/foo?pretty
curl -XPUT 127.0.0.1:9200/foo/doc/1?pretty -d '{ "foo": "bar" }'
curl -XPUT "127.0.0.1:9200/_snapshot/my_s3_repository/snapshot_1?wait_for_completion=true&pretty"

This is working for me on a one node cluster with no docker layer involved.

For the record, I tried with a 2 nodes cluster and it went well.

Thanks for replying.

I tried as you mentioned, but still got same error.

curl -X PUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -d '{
  "type": "s3",
    "settings": {
      "bucket": "my.bucket",
      "region": "ap-northeast-1"
    }
}'

{
  "error" : {
    "root_cause" : [ {
      "type" : "repository_verification_exception",
      "reason" : "[my_s3_repository] path  is not accessible on master node"
    } ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-R455L68wQiSdCB0hhjuUaQ/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D7A94303C3A9214)"
      }
    }
  },
  "status" : 500
}

Would it be possible to try with this?

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

Should I state it as IAM policy, not Bucket policy?

Yes. Read https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-repository.html#cloud-aws-repository-permissions

This may be configured through the AWS IAM console, by creating a Custom Policy, and using a Policy Document similar to this (changing snaps.example.com to your bucket name).

Thanks.
I have already given s3 full access to the ecs task roll.

However, after adding everyone to bucket policy acl, all processes went well.
And according to IAM history, the S3FullAccess policy was not used.

So I guessed that the aws-cloud-plugin is not using sdk, is it right ?

It is using AWS Java SDK.

I thought AWS cloud plugin needs only IAM role, which is assigned to the instance that ElasticSearch is running on, to communicate with S3.
So, I just gave S3FullAccess to ECS task role and permitted the role in S3 bucket policy.

But actually, AWS cloud plugin needs user credential keys.
I must give the credentials to the plugin, and the credentials must be assigned to the user who has S3 access authority.

These my understandings are correct?
Sorry for my odd English.

I don't think you must set key/secret. If you are running on AWS, using a policy IAM thing should be ok.
This is how we are initializing the client: https://github.com/elastic/elasticsearch/blob/2.4/plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java#L135-L145

    if (account == null && key == null) {
        credentials = new AWSCredentialsProviderChain(
                new EnvironmentVariableCredentialsProvider(),
                new SystemPropertiesCredentialsProvider(),
                new InstanceProfileCredentialsProvider()
        );
    } else {
        credentials = new AWSCredentialsProviderChain(
                new StaticCredentialsProvider(new BasicAWSCredentials(account, key))
        );
    }

So you have multiple options available here.

Perfectly fine and clear. English is not my native language either. :slight_smile:

Thanks,

I understand aws-cloud-plugin doesn't necessarily need a credential key set.

But, I think aws-cloud-plugin is not using ECS task role in my case.

I confirmed ECS task role was enabled by below process.

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
In docker container which ElasticSearch running on, the command "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" returns ECS task role.

http://docs.aws.amazon.com/cli/latest/reference/sts/get-caller-identity.html
I also installed aws-sdk to same container and executed get-caller-identity.
It returned ECS task role.

So, It can be said ECS task role was enabled.

As I wrote firstly, when I gave S3FullAccess to ECS task role, the snapshot process failed.
But when I gave same authority to IAM role which is belongs to ECS instance, the process went well.
Bucket policy permits the access from account root, so I think aws-cloud-plugin is not using ECS task role, but IAM role of ECS instance.

In your quote, maybe, the InstanceProfileCredentialsProvider handles IAM role and I feel this class doesn't care ECS task role from the name. (Sorry for not reading actual codes.)

Ok. I missed actually that you were running with ECS. I thought you were using EC2.
I have absolutely no experience with ECS and I'm unsure if we can "fix" anything on our end.

May be updating the AWS SDK to a more recent version could help cloud-aws plugin to use ECS task role but again I have no clue here.

OK. Thanks.