Getting AWS credentials from keystore for discovery-ec2 plugin?

We are upgrading to ES 5.4.x after having used 1.7.3 and 2.3.3 in production. Having learned that including AWS credentials in config/elasticsearch.yml has been discouraged, I setup a keystore to hold those settings. But it seems the discovery-ec2 plugin is not using that resource to obtain credentials. Am I missing a setting somewhere to allow that behavior?

It should work OOTB. What exactly did you configure in the keystore? Can you list the settings you have?

I added these settings: (command and output)
./elasticsearch-keystore list

Using chef to provision the host, executed commands as follows:
#{es_home_dir}/bin/elasticsearch-keystore create
/bin/echo #{es_aws_access_key} | #{es_home_dir}/bin/elasticsearch-keystore add --stdin
/bin/echo #{es_aws_secret_access_key} | #{es_home_dir}/bin/elasticsearch-keystore add --stdin
/bin/echo 'us-east-1' | #{es_home_dir}/bin/elasticsearch-keystore add --stdin

When I issue the list command, should I also see the values with the keys?

I see. Sadly the documentation has not been updated yet. It needs to be:

  • discovery.ec2.access_key
  • discovery.ec2.secret_key should not be used anymore but discovery.ec2.endpoint needs to be set in elasticsearch.yml file.


I hope this helps.

cc @rjernst in case I'm saying something wrong :slight_smile:

Unfortunately, changing keystore names did not solve the issue. We have solved the problem by entering the updated key names in elasticsearch.yml.

@Jaff What do you mean by We have solved the problem by entering the updated key names in elasticsearch.yml? David is correct that the keystore settings should be discovery.ec2.access_key and discovery.ec2.secret_key. Any of the old settings will not work there (unfortunately we do not yet have validation of settings in the keystore, so you will not get an error if you add settings that should not be there). But setting eg discovery.ec2.access_key in elasticsearch.yml should not work. This setting should only be allowed in the keystore.

We are OK with discovery-ec2 plugin, however repository-s3 authentication is failing. I have set keystore values for and and still getting failure.

Ok, repository-s3 has different settings. This is due to that plugin allow for multiple credential sets. There you need to set s3.client.default.access_key and s3.client.default.secret_key.

1 Like

The different keystore settings did affect the result, now able to create bucket. Here is keystore list output:
/mnt/elasticsearch# bin/elasticsearch-keystore list

Here is result of running shell script to create the repository bucket:
/mnt/elasticsearch# bin/ es-lineartv_test

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.