Can I omit ES_USERNAME, ES_PASSWORD, KIBANA_FLEET_USERNAME, KIBANA_FLEET_PASSWORD from elastic-agent Kubernetes daemonset?

Elastic Stack Elastic Agent
1

In the Fleet UI, when specifying the installation method for a Kubernetes cluster,
ES_USERNAME and ES_PASSWORD are specified as environment variables which are
passed down to the docker.elastic.co/beats/elastic-agent container here:

Is setting ES_USERNAME and ES_PASSWORD mandatory,
or can these be omitted, with elastic-agent obtaining these values automatically from Fleet?

The reason I am asking is because:

  1. I have about 10 Kubernetes clusters with the elastic-agent daemonset installed
  2. I want to periodically rotate the Elasticsearch elastic user's password
  3. I don't want to go through the headache of updating the ES_PASSWORD value in each Kubernetes cluster
  4. I think I have some elastic-agent's with a misconfigured because if I look at the traffic to Elasticsearch, I am right now seeing a lot of HTTP 401 errors
  5. I'd rather that Elastic Agent get the credentials it needs to access Elasticsearch via Fleet and Kibana
2

OK, so I did some more investigation, and after reading:

I found that there are two separate Kubernetes manifests:

The Fleet managed manifest does not specify ES_USERNAME.

The Standalone manifest does specify ES_USERNAME.

However the Fleet managed manifest does have KIBANA_FLEET_USERNAME and KIBANA_FLEET_PASSWORD.

Do I really need those, or can I omit them?

I really don't want to maintain credentials to log into Fleet in this file.

I also see that https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/elastic_agent_manifest.ts

has two separate manifests elasticAgentStandaloneManifest and elasticAgentManagedManifest.

The different manifests are selected based on if in the Fleet UI you select Enroll in Fleet or Run standalone

3

Also, if I look at this comment:

            # Elasticsearch API key used to enroll Elastic Agents in Fleet (https://www.elastic.co/guide/en/fleet/current/fleet-enrollment-tokens.html#fleet-enrollment-tokens)
            # If FLEET_ENROLLMENT_TOKEN is empty then KIBANA_HOST, KIBANA_FLEET_USERNAME, KIBANA_FLEET_PASSWORD are needed
            - name: FLEET_ENROLLMENT_TOKEN

So based on that, I assume that if I specify:

FLEET_SERVER
FLEET_ENROLLMENT_TOKEN

then I can omit:

KIBANA_HOST
KIBANA_FLEET_USERNAME
KIBANA_FLEET_PASSWORD

It looks like this is the logic in Elastic Agent which initializes:

defaultAccessConfig()

but it is not clear to me if KIBANA_FLEET_USERNAME can be omitted, since defaultAccessConfig() populates that value with a default.

However in newContainerCommand there is a big comment which explains more:

The following actions are possible and grouped based on the actions.

* Elastic Agent Fleet Enrollment
  This enrolls the Elastic Agent into a Fleet Server. It is also possible to have this create a new enrollment token
  for this specific Elastic Agent.

  FLEET_ENROLL - set to 1 for enrollment into Fleet Server. If not set, Elastic Agent is run in standalone mode.
  FLEET_URL - URL of the Fleet Server to enroll into
  FLEET_ENROLLMENT_TOKEN - token to use for enrollment. This is not needed in case FLEET_SERVER_ENABLED and FLEET_ENROLL is set. Then the token is fetched from Kibana.
  FLEET_CA - path to certificate authority to use with communicate with Fleet Server [$KIBANA_CA]
  FLEET_INSECURE - communicate with Fleet with either insecure HTTP or unverified HTTPS


  The following vars are need in the scenario that Elastic Agent should automatically fetch its own token.

  KIBANA_FLEET_HOST - Kibana host to enable create enrollment token on [$KIBANA_HOST]
  FLEET_TOKEN_NAME - token name to use for fetching token from Kibana. This requires Kibana configs to be set.
  FLEET_TOKEN_POLICY_NAME - token policy name to use for fetching token from Kibana. This requires Kibana configs to be set.

* Bootstrapping Fleet Server
  This bootstraps the Fleet Server to be run by this Elastic Agent. At least one Fleet Server is required in a Fleet
  deployment for other Elastic Agents to bootstrap. In case the Elastic Agent is run without Fleet Server, these variables
  are not needed.

  If FLEET_SERVER_ENABLE and FLEET_ENROLL is set but no FLEET_ENROLLMENT_TOKEN, the token is automatically fetched from Kibana.

  FLEET_SERVER_ENABLE - set to 1 enables bootstrapping of Fleet Server inside Elastic Agent (forces FLEET_ENROLL enabled)
  FLEET_SERVER_ELASTICSEARCH_HOST - Elasticsearch host for Fleet Server to communicate with [$ELASTICSEARCH_HOST]
  FLEET_SERVER_ELASTICSEARCH_CA - path to certificate authority to use to communicate with Elasticsearch [$ELASTICSEARCH_CA]
  FLEET_SERVER_ELASTICSEARCH_CA_TRUSTED_FINGERPRINT - The sha-256 fingerprint value of the certificate authority to trust
  FLEET_SERVER_ELASTICSEARCH_INSECURE - disables cert validation for communication with Elasticsearch
  FLEET_SERVER_SERVICE_TOKEN - service token to use for communication with Elasticsearch
  FLEET_SERVER_SERVICE_TOKEN_PATH - path to service token file to use for communication with Elasticsearch
  FLEET_SERVER_POLICY_ID - policy ID for Fleet Server to use for itself ("Default Fleet Server policy" used when undefined)
  FLEET_SERVER_HOST - binding host for Fleet Server HTTP (overrides the policy). By default this is 0.0.0.0.
  FLEET_SERVER_PORT - binding port for Fleet Server HTTP (overrides the policy)
  FLEET_SERVER_CERT - path to certificate to use for HTTPS endpoint
  FLEET_SERVER_CERT_KEY - path to private key for certificate to use for HTTPS endpoint
  FLEET_SERVER_CERT_KEY_PASSPHRASE - path to private key passphrase file for certificate to use for HTTPS endpoint
  FLEET_SERVER_INSECURE_HTTP - expose Fleet Server over HTTP (not recommended; insecure)
  FLEET_SERVER_INIT_TIMEOUT - Sets the initial timeout when starting up the fleet server under agent. Default: 30s.

* Preparing Kibana for Fleet
  This prepares the Fleet plugin that exists inside of Kibana. This must either be enabled here or done externally
  before Fleet Server will actually successfully start. All the Kibana variables are not needed in case Elastic Agent
  should not setup Fleet.

  KIBANA_FLEET_HOST - Kibana host accessible from Fleet Server. [$KIBANA_HOST]
  KIBANA_FLEET_USERNAME - Kibana username to service token [$KIBANA_USERNAME]
  KIBANA_FLEET_PASSWORD - Kibana password to service token [$KIBANA_PASSWORD]
  KIBANA_FLEET_CA - path to certificate authority to use with communicate with Kibana [$KIBANA_CA]
  KIBANA_REQUEST_RETRY_SLEEP - sleep duration taken when agent performs a request to Kibana [default 1s]
  KIBANA_REQUEST_RETRY_COUNT - number of retries agent performs when executing a request to Kibana [default 30]

The following environment variables are provided as a convenience to prevent a large number of environment variables to
be used when the same credentials will be used across all the possible actions above.

  ELASTICSEARCH_HOST - Elasticsearch host [http://elasticsearch:9200]
  ELASTICSEARCH_USERNAME - Elasticsearch username [elastic]
  ELASTICSEARCH_PASSWORD - Elasticsearch password [changeme]
  ELASTICSEARCH_CA - path to certificate authority to use to communicate with Elasticsearch
  KIBANA_HOST - Kibana host [http://kibana:5601]
  KIBANA_FLEET_USERNAME - Kibana username to enable Fleet [$ELASTICSEARCH_USERNAME]
  KIBANA_FLEET_PASSWORD - Kibana password to enable Fleet [$ELASTICSEARCH_PASSWORD]
  KIBANA_CA - path to certificate authority to use with communicate with Kibana [$ELASTICSEARCH_CA]
  ELASTIC_AGENT_TAGS - user provided tags for the agent [linux,staging]


By default when this command starts it will check for an existing fleet.yml. If that file already exists then
all the above actions will be skipped, because the Elastic Agent has already been enrolled. To ensure that enrollment
occurs on every start of the container set FLEET_FORCE to 1.
4

@Nicolas_Ruflin @ruflin Can you shed any light on this topic?
It looks like you submitted a big patch to the fleet server docs in Update docs around Elastic Agent and fleet-server environment variables by ruflin · Pull Request #24778 · elastic/beats · GitHub

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.