Registering S3 repository to ElasticSearch running on AWS ECS

111131 · September 27, 2016, 5:05am

HI.

I'm trying to register S3 repository to ES(v.2.4) which is docker container running on AWS ECS cluster instance.

config/elasticsearch.yml

 repositories:
      s3:
        bucket: "my.bucket"
        region: "ap-northeast-1"

S3 bucket policy

   {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:ListBucket",
    				"s3:GetBucketLocation",
    				"s3:ListBucketMultipartUploads",
    				"s3:ListBucketVersions"
    			],
    			"Resource": "arn:aws:s3:::my.bucket"
    		},
    		{
    			"Effect": "Allow",
    			"Principal": {
    				"AWS": "arn:aws:iam::role/ecs-task-role"
    			},
    			"Action": [
    				"s3:GetObject",
    				"s3:PutObject",
    				"s3:DeleteObject",
    				"s3:AbortMultipartUpload",
    				"s3:ListMultipartUploadParts"
    			],
    			"Resource": "arn:aws:s3:::my.bucket/*"
    		}
    	]
    }

I gave S3FullAccess to ECS task roll.

GET _nodes/?pretty
    ...
    "repositories": {
              "s3": {
                "bucket": "my.bucket",
                "region": "ap-northeast-1"
              }
    ...

GET _snapshot/?pretty
{}

I got an error like this, when I tried to register repository.

PUT _snapshot/s3_snapshot_repository
    {
      "type": "s3",
      "settings": {
        "bucket": "my.bucket",
        "region": "ap-northeast-1"
      }
    }
{
   "error": {
      "root_cause": [
         {
            "type": "repository_verification_exception",
            "reason": "[s3_snapshot_repository] path  is not accessible on master node"
         }
      ],
      "type": "repository_verification_exception",
      "reason": "[s3_snapshot_repository] path  is not accessible on master node",
      "caused_by": {
         "type": "i_o_exception",
         "reason": "Unable to upload object tests-Xa3F_VHATDuJ45kz5jo6rg/master.dat-temp",
         "caused_by": {
            "type": "amazon_s3_exception",
            "reason": "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied"
         }
      }
   },
   "status": 500
}

However, test files were generated to S3 bucket.

What should I do to solve this error?

dadoonet · September 27, 2016, 5:52am

Which version?

111131 · September 27, 2016, 5:55am

Sorry, I forgot to mention about it.
I'm using version 2.4.

dadoonet · September 27, 2016, 6:30am

Thanks! I had another similar report on 2.4 so I'll try to reproduce and come back later.

dadoonet · September 27, 2016, 7:29am

Can you try a similar test without docker? Like: start an EC2 instance and run something like:

wget https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.4.0/elasticsearch-2.4.0.tar.gz
tar xzf elasticsearch-2.4.0.tar.gz 
cd elasticsearch-2.4.0
bin/plugin install cloud-aws
vi config/elasticsearch.yml

Then add your cloud settings:

cloud.aws.access_key: "KEY"
cloud.aws.secret_key: "SECRET"
cloud.aws.region: REGION

Then launch:

bin/elasticsearch

In another terminal:

curl -XDELETE 127.0.0.1:9200/_snapshot/my_s3_repository?pretty
curl -XPUT '127.0.0.1:9200/_snapshot/my_s3_repository?pretty' -d'
{
  "type": "s3",
  "settings": {
    "bucket": "your-bucket-here",
    "region": "your-region-here"
  }
}'
curl -XDELETE 127.0.0.1:9200/foo?pretty
curl -XPUT 127.0.0.1:9200/foo/doc/1?pretty -d '{ "foo": "bar" }'
curl -XPUT "127.0.0.1:9200/_snapshot/my_s3_repository/snapshot_1?wait_for_completion=true&pretty"

This is working for me on a one node cluster with no docker layer involved.

dadoonet · September 27, 2016, 7:42am

For the record, I tried with a 2 nodes cluster and it went well.

111131 · September 27, 2016, 9:29am

Thanks for replying.

I tried as you mentioned, but still got same error.

curl -X PUT 'localhost:9200/_snapshot/my_s3_repository?pretty' -d '{
  "type": "s3",
    "settings": {
      "bucket": "my.bucket",
      "region": "ap-northeast-1"
    }
}'

{
  "error" : {
    "root_cause" : [ {
      "type" : "repository_verification_exception",
      "reason" : "[my_s3_repository] path  is not accessible on master node"
    } ],
    "type" : "repository_verification_exception",
    "reason" : "[my_s3_repository] path  is not accessible on master node",
    "caused_by" : {
      "type" : "i_o_exception",
      "reason" : "Unable to upload object tests-R455L68wQiSdCB0hhjuUaQ/master.dat-temp",
      "caused_by" : {
        "type" : "amazon_s3_exception",
        "reason" : "Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 8D7A94303C3A9214)"
      }
    }
  },
  "status" : 500
}

dadoonet · September 27, 2016, 10:07am

Would it be possible to try with this?

{
  "Statement": [
    {
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListBucketVersions"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket"
      ]
    },
    {
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload",
        "s3:ListMultipartUploadParts"
      ],
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::my.bucket/*"
      ]
    }
  ],
  "Version": "2012-10-17"
}

111131 · September 27, 2016, 10:24am

Should I state it as IAM policy, not Bucket policy?

dadoonet · September 27, 2016, 10:39am

Yes. Read https://www.elastic.co/guide/en/elasticsearch/plugins/current/cloud-aws-repository.html#cloud-aws-repository-permissions

This may be configured through the AWS IAM console, by creating a Custom Policy, and using a Policy Document similar to this (changing snaps.example.com to your bucket name).

111131 · September 28, 2016, 4:30am

Thanks.
I have already given s3 full access to the ecs task roll.

However, after adding everyone to bucket policy acl, all processes went well.
And according to IAM history, the S3FullAccess policy was not used.

So I guessed that the aws-cloud-plugin is not using sdk, is it right ?

dadoonet · September 28, 2016, 5:21am

It is using AWS Java SDK.

111131 · September 28, 2016, 5:46am

I thought AWS cloud plugin needs only IAM role, which is assigned to the instance that ElasticSearch is running on, to communicate with S3.
So, I just gave S3FullAccess to ECS task role and permitted the role in S3 bucket policy.

But actually, AWS cloud plugin needs user credential keys.
I must give the credentials to the plugin, and the credentials must be assigned to the user who has S3 access authority.

These my understandings are correct?
Sorry for my odd English.

dadoonet · September 28, 2016, 10:55am

I don't think you must set key/secret. If you are running on AWS, using a policy IAM thing should be ok.
This is how we are initializing the client: https://github.com/elastic/elasticsearch/blob/2.4/plugins/cloud-aws/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java#L135-L145

    if (account == null && key == null) {
        credentials = new AWSCredentialsProviderChain(
                new EnvironmentVariableCredentialsProvider(),
                new SystemPropertiesCredentialsProvider(),
                new InstanceProfileCredentialsProvider()
        );
    } else {
        credentials = new AWSCredentialsProviderChain(
                new StaticCredentialsProvider(new BasicAWSCredentials(account, key))
        );
    }

So you have multiple options available here.

Perfectly fine and clear. English is not my native language either.

111131 · September 29, 2016, 8:53am

Thanks,

I understand aws-cloud-plugin doesn't necessarily need a credential key set.

But, I think aws-cloud-plugin is not using ECS task role in my case.

I confirmed ECS task role was enabled by below process.

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html
In docker container which ElasticSearch running on, the command "curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" returns ECS task role.

http://docs.aws.amazon.com/cli/latest/reference/sts/get-caller-identity.html
I also installed aws-sdk to same container and executed get-caller-identity.
It returned ECS task role.

So, It can be said ECS task role was enabled.

As I wrote firstly, when I gave S3FullAccess to ECS task role, the snapshot process failed.
But when I gave same authority to IAM role which is belongs to ECS instance, the process went well.
Bucket policy permits the access from account root, so I think aws-cloud-plugin is not using ECS task role, but IAM role of ECS instance.

In your quote, maybe, the InstanceProfileCredentialsProvider handles IAM role and I feel this class doesn't care ECS task role from the name. (Sorry for not reading actual codes.)

dadoonet · September 30, 2016, 10:17am

Ok. I missed actually that you were running with ECS. I thought you were using EC2.
I have absolutely no experience with ECS and I'm unsure if we can "fix" anything on our end.

May be updating the AWS SDK to a more recent version could help cloud-aws plugin to use ECS task role but again I have no clue here.

111131 · September 30, 2016, 10:21am

OK. Thanks.

Topic		Replies	Views
Can't register snapshot repository in elasticsearch Elasticsearch	1	1444	August 20, 2018
Can't able register aws ES snapshot repository to filesystem Elasticsearch	2	1165	February 14, 2019
Can't register snapshot repository to S3 Elasticsearch	1	790	September 10, 2019
Can't seem to register a snapshot repository with AWS S3 Elasticsearch	2	741	March 23, 2019
Snapshot repository-s3 registry failed in 7.17 Elasticsearch docker , snapshot-and-restore	5	246	May 30, 2024

Registering S3 repository to ElasticSearch running on AWS ECS

Related topics