Metricbeat with namespace secrets in Kubernetes

Elastic Stack Beats
1

I have metricbeat up and running on kubernetes collecting k8s metrics, as well as node metrics, forwarding to ES, deployed as a deamonset via helm chart.

I want to use some of the other modules (rabbitmq, redis, etc), but the secrets for those are in a different namespace. What is the recommended way of enabling metricbeat to capture those additional metrics?

2

Hi @soandosnc and welcome :slight_smile:

It is not possible at the moment to use Kubernetes secrets with Beats. There is an open issue to support that: https://github.com/elastic/beats/issues/8847

There would be some alternatives, but they wouldn't be as secure as using secrets.
Take a look to hints-based autodiscover, that allows to setup the configuration, including credentials, in the annotations of the pods, but this would require to copy the secrets to the pod annotations, what can be unsafe.
You could also mount the secrets as environment variables in the metricbeat pod and use them in the autodiscover templates, but this would require to copy the secrets into the namespace where metricbeat is deployed, and any change on these secrets would need to redeploy metricbeat.

3

Can we do better with per namespace deployments of metric beats?

I would wonder if there is a way to mount the secrets, and then every some unit time restart the metric beat pod (maybe doing a rolling upgrade so there is no loss of coverage, though at least in my case a gap of a few seconds is acceptable).

If there are multiple namespaces then you would need multiple metric beats pods, but you would probably want different kibana boards in any case for that.

What I'm not clear on is how to do the service discovery. I see that the hint autodiscovery can get the secrets, but if I don't want to actually put the secret in the annotation, is there a way to put the secret key and value name in the annotation, and then
have metricbeat get the secret from k8s? I don't think there is a technical problem, but I also don't know that it's supported.

Thanks for the warm welcome

4

Yes, using a metricbeat per namespace would be another option, this way you could mount the secrets as environment variables in the metricbeat pod. If you do it, remember to set the namespace setting to the proper namespace in the kubernetes settings of each metricbeat.

Yeah, at the moment it would be needed to copy the secrets to the annotations, what is not very nice. The open issue is to avoid the need of doing that.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.