ECE Elasticsearch Keystore

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I am trying to use the elasticsearch keystore feature in ECE however I am having a few issues. First issue, when simply adding a simple keystore key value pair it will cause instance boot looping on all future stack changes. Seems strange but that is the first order of business to resolve.

Second, after the boot looping issue is resolved I will need to understand how I can use the keystore item as the instrucure are very vague and don't provide any example on how to use the keystore secrets. (Example below)

xpack.monitoring.exporters.my_remote.auth.password: "MY HARD CODED PASSWORD"

vs 

xpack.monitoring.exporters.my_remote.auth.password: "${MY_PWD}"

ECE Keystore Documentation
https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configuring-keystore.html

2

when simply adding a simple keystore key value pair it will cause instance boot looping on all future stack changes

This is a known limitation of the keystore currently - what it means is that you mis-configured the setting in the keystore .. the problem is that the validation doesn't occur on each instance until it is (re)started

When it bootloops, you should be able to look in the L+M cluster and see the specific config error causing the problem

We appreciate that this is far from ideal, and there's a lot of discussion going on about how to improve it.

Second, after the boot looping issue is resolved I will need to understand how I can use the keystore item

I may have misunderstood the question, but you use the keystore when explicitly directed to by the ES docs (ie a specific parameter is marked as "secure"), at which point in the ECE keystore UI you enter the full path for that parameter as the "key" and a hard-coded string (no env vars) as the "value" ... does that help?

3

Okay, let me rephrase my questions for what my end goal is to make sure we are on the same page.

I need the ability to keep the values below secure in my elasticserach.yml file in ECE by not entering hard coded passwords.

Questions

  1. Is this possible?
  2. If it is possible how is it accomplished? (Example by Elasticsearch Keystore similar to the logstash keystore or some other means)

Secure Configuration Example (Please note the secure references by ${MY_SECURE_CONTENT})

xpack.monitoring.exporters.my_remote.auth.password: "${MY_PWD}"

xpack.monitoring.elasticsearch:
  hosts: ["https://URL:443"]
  username: "${MY_USER}"
  password: "${MY_PWD}"
4

Unfortunately

I need the ability to keep the values below secure in my elasticserach.yml file in ECE by not entering hard coded passwords.

Is not possible. The most secure you can get is by entering fields marked as "secure" in the ES docs into the keystore UI, which is very marginally more secure than entering them directly into the YAML config box.

The credentials will still be on disk in accessible form inside the container, and some (sufficiently authorized) API calls will still leak them

It sounds like what you want is to be able to specify env variables at the host level and have ECE pick those up - this is something under discussion as we improve the keystore security, but not currently possible

Alex

5

Thank you for your quick reply!

Hopefully this is in the development roadmap pipeline to provide secure keystore functionality very similar to the Logstash keystore where I can reference any custom made keystore value in my yml configuration. It is generally a bad idea to have hardcoded passwords in yml files as we need the ability to import the files into our Github repo for CI/CD pipeline automation.

Logstash Keystore
https://www.elastic.co/guide/en/logstash/current/keystore.html

6

It is generally a bad idea to have hardcoded passwords in yml files as we need the ability to import the files into our Github repo for CI/CD pipeline automation

Wait which YAML files are you pulling from ECE?

The only "secure" credentials that should appear in the (internally generated/managed) ES YAML files are legacy parameters that haven't been migrated to the keystore yet (I think the monitoring password is one of them actually)

So in terms of the stack - the long term plan is not to make the keystore accept arbitrary strings, it's just to all port "secure" fields to the keystore. Then ECE provides an API for adding those fields to the keystore. So far so good.

The problem is that ECE currently does not do a great job of isolating those secure parameters - they are on disk in a different internal file, and also some root-level APIs will "leak" them .. this is the area we are (re)designing at the moment

Alex

7

I need the ability to forward monitoring data to a separate ECE cluster. I have to setup the configuration below in ECE for the elasticsearch.yml and kibana.yml files. I have been able to accomplish this with the settings below however they are currently not secure as the password data is in clear text.

I am simply trying to secure the hard coded passwords in the ECE elasticserach.yml and kibana.yml config file as noted below.

elasticsearch.yml

xpack.monitoring.enabled: true

xpack.monitoring.elasticsearch.collection.enabled: true

xpack.monitoring.collection.enabled: true

xpack.monitoring.exporters.my_remote.type: http
xpack.monitoring.exporters.my_remote.host: [ "https://URL:443" ]
xpack.monitoring.exporters.my_remote.auth.username: "**HARD CODED USER**"
xpack.monitoring.exporters.my_remote.auth.password: "**HARD CODED PASSWORD**"

kibana.yml

xpack.monitoring.enabled: true
xpack.monitoring.kibana.collection.enabled: true
xpack.monitoring.ui.enabled: true

xpack.monitoring.elasticsearch:
  hosts: ["https://URL:443"]
  username: "**HARD CODED USER**"
  password: "**HARD CODED PASSWORD**"
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.