Configuring Fleet-managed Elastic Agent on ECK with external Fleet

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

So I'm wondering what's the official recommended way to configure a Fleet-managed Elastic Agent on ECK that connects to a Fleet server that is not within the same Kubernetes cluster. I'd assume this is a pretty common case (maybe even more common than having the clients and server in the same cluster), yet I didn't find any explicit documentation.

In reality I did manage to set it up using the ENV variables recommended here, but I do have some doubts/questions:

  1. Is the the canonical way of doing this? (If so, I think a more explicit documentation would be welcome.)

  2. If using this method the operator gives me a warning when applying the manifest: "spec.PolicyID is empty, spec.PolicyID will become mandatory in a future release". This makes me doubt the solution a bit.

  3. (Loosely related question) To avoid having verification issues, I'm using a certificate signed by a "real" CA, and have it set up for ES, Kibana and Fleet the way it's documented here. It works, however now for internal agents I have to set the FLEET_INSECURE env, and for the server itself I also have to set the undocumented FLEET_SERVER_ELASTICSEARCH_INSECURE. (No surprise here, the certificate does not [cannot] include the internal service names.)

    Couldn't/shouldn't the operator take care of these? It would be consistent with Kibana, which decreases the verification mode so it has no issues with the ES certificate.

  4. (Even more loosely related question) Out of curiosity, when the agent is "internal", why is a kibanaRef required, when for "external" cases the fleet server address is sufficient for the enrollment.

2

Hey @lpeter,

Thank you for your interest in ECK and Fleet!

When components aren’t fully contained within an ECK cluster, some configuration needs to be handled manually. For Fleet/Agent, this involves setting environment variables accordingly.

Looking ahead, I believe the PolicyID will likely become mandatory, regardless of where the different components are located.

To assist with setting up trust, someone recently contributed documentation that could be particularly useful for your situation: How to Register an Agent Outside of ECK. It would be ideal to have similar documentation directly within ECK; we’ll explore how this fits into our upcoming roadmap.

When an agent operates internally, ECK retrieves the enrollment token directly from Kibana, which could explain why a kibanaRef is necessary in those cases.

3

Hi!

Thank you for your input (and sorry for the late reply).

I'm happy with my current solution without ingresses, I think it's a bit less configuration, but one could argue that using an ingress is the more "K8s way" of doing things. :slight_smile:

As for the mandatory PolicyID, I'm unsure if that's the right path ahead, I believe the enrollment token also carries authentication/authorization data, which the policy ID does not. So it would require additional authnz data that provides the required Kibana access to fetch the actual enrollment token. Ideally this access would be limited to fetching a single token [1], which would render the PolicyID useless yet again.

[1] Giving endpoints the ability to enroll into arbitrary policies is probably bad design. Especially if the Fleet server and endpoint Agents are operated by different teams.