Contact an APMServer from several namespaces

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello,

I've recently been trying to deploy a single ApmServer for several namespace. Unfortunately, it means that I have to manually copy the apm-token to other namespace that might need it.
It makes token rotations time-consuming and a bit risky (for instance, What if I forgot one token secret?)
I thought about tasking the operator to create the secrets for me, however this would lead to an invalid ownerReferences as it is only possible to reference cluster-scoped resources or resources in the same namespace.

Which is why I thought it was a good idea to have one ApmServer per namespace. Unfortunately, this is currently not supported by the operator as the current documentation indicate:

Deploying the APM Server and Elasticsearch in two different namespaces is currently not supported.

How do you guys handle having agents in all namespaces? Do you manually copy the secret? Do you manually specify a secret to the operator so that you can handle rotations yourself? Do you disable authentication all-together?

Best regards,

2

Hey @Sayrus,

Deploying the APM Server and Elasticsearch in two different namespaces is currently not supported.

Oops, this isn't true anymore. We definitely support running Elasticsearch and APMServer in different namespaces, and we already supported it in 1.0. I'm fixing the docs to remove that warning.

When copying Secrets into other namespace you need to be extra careful about this Kubernetes bug: you must make sure you don't copy the metadata.ownerReferences field.

This issue in the cert-manager repo contains some examples of CronJobs to deal with the synchronization of Secrets in multiple namespaces.
I am not aware of a builtin solution or simple operator you could use for that purpose.

Simplifying secret management across namespaces is definitely something we (ECK team) are thinking about.

3

Thanks for this precise answer. This is great news, I'll be able to achieve what I wanted by having an APMServer in each namespace. As long as I am not using Agent Autoconfiguration, I don't even need secret synchronization because ECK will handle the ES required secrets and the APMServer has its own.

The issue you linked are indeed very serious and I'll be taking it into account even outside ECK.

I just tried deploying an APMServer in another namespace and it work like a charm.

Thanks again,