Getting AWS credentials from keystore for discovery-ec2 plugin?


(Mark Jaffe) #1

We are upgrading to ES 5.4.x after having used 1.7.3 and 2.3.3 in production. Having learned that including AWS credentials in config/elasticsearch.yml has been discouraged, I setup a keystore to hold those settings. But it seems the discovery-ec2 plugin is not using that resource to obtain credentials. Am I missing a setting somewhere to allow that behavior?


(David Pilato) #2

It should work OOTB. What exactly did you configure in the keystore? Can you list the settings you have?


(Mark Jaffe) #3

I added these settings: (command and output)
./elasticsearch-keystore list
cloud.aws.access_key
cloud.aws.region
cloud.aws.secret_key

Using chef to provision the host, executed commands as follows:
#{es_home_dir}/bin/elasticsearch-keystore create
/bin/echo #{es_aws_access_key} | #{es_home_dir}/bin/elasticsearch-keystore add --stdin cloud.aws.access_key
/bin/echo #{es_aws_secret_access_key} | #{es_home_dir}/bin/elasticsearch-keystore add --stdin cloud.aws.secret_key
/bin/echo 'us-east-1' | #{es_home_dir}/bin/elasticsearch-keystore add --stdin cloud.aws.region


(Mark Jaffe) #4

When I issue the list command, should I also see the values with the keys?


(David Pilato) #5

I see. Sadly the documentation has not been updated yet. It needs to be:

  • discovery.ec2.access_key
  • discovery.ec2.secret_key

cloud.aws.region should not be used anymore but discovery.ec2.endpoint needs to be set in elasticsearch.yml file.

See https://github.com/elastic/elasticsearch/blob/5.4/plugins/discovery-ec2/src/main/java/org/elasticsearch/discovery/ec2/AwsEc2Service.java#L181-L188

I hope this helps.

cc @rjernst in case I'm saying something wrong :slight_smile:


S3-repository deprecated settings warning
Elasticsearch.yml chanages 2.x to 5.x on AWS EC2 - Amazon linux
Elasticsearch.yml chanages 2.x to 5.x on AWS EC2 - Amazon linux
(Mark Jaffe) #6

Unfortunately, changing keystore names did not solve the issue. We have solved the problem by entering the updated key names in elasticsearch.yml.


(Ryan Ernst) #7

@Jaff What do you mean by We have solved the problem by entering the updated key names in elasticsearch.yml? David is correct that the keystore settings should be discovery.ec2.access_key and discovery.ec2.secret_key. Any of the old settings will not work there (unfortunately we do not yet have validation of settings in the keystore, so you will not get an error if you add settings that should not be there). But setting eg discovery.ec2.access_key in elasticsearch.yml should not work. This setting should only be allowed in the keystore.


(Mark Jaffe) #8

We are OK with discovery-ec2 plugin, however repository-s3 authentication is failing. I have set keystore values for cloud.aws.s3.access_key and cloud.aws.s3.secret_key and still getting failure.


(Ryan Ernst) #9

Ok, repository-s3 has different settings. This is due to that plugin allow for multiple credential sets. There you need to set s3.client.default.access_key and s3.client.default.secret_key.


(Mark Jaffe) #11

The different keystore settings did affect the result, now able to create bucket. Here is keystore list output:
/mnt/elasticsearch# bin/elasticsearch-keystore list
discovery.ec2.access_key
discovery.ec2.secret_key
s3.client.default.access_key
s3.client.default.secret_key

Here is result of running shell script to create the repository bucket:
/mnt/elasticsearch# bin/s3-repo-create.sh es-lineartv_test
{"acknowledged":true}


S3-repository deprecated settings warning
(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.