How to user docker secrets with metricbeat

[denisk (Denis)]

I would like to use docker secrets [1] to provide credentials for metricbeat to use when running the monitoring modules. Is that possible? I am using the docker image of Metricbeat 8.11.3 from Docker hub.

I know metricbeat supports creating a keystore [2], however that forces me to start the container, before I can configure the credentials. And I would need to do that for every environment manually again.

I would like to reuse the same metricbeat docker container in different environment with the same monitoring setup, and different credentials (e.g. test/acc/prod).
I was hoping to be able to use docker secrets for that.

My docker-compose file is below. The account credentials that I would need to monitor one of my services (in this case activemq) would then be in secret_stuff.properties

---
version: '3.8'
name: my_setup

secrets:
  activemq_credentials:
    file: secret_stuff.properties
    
services:
  kiara-metricbeat:
    image: my_metricbeat_image
    user: root
    secrets:
      - activemq_credentials    
    volumes:
      ...

  activemq:
    image: my-activemq   
    ports:
      - "8161:8161"

In the metricbeat image I added the configuration for monitoring the activemq container:

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      templates:
        - condition:
            contains:
              container.image.name: "activemq"
          config:
            - module: activemq
              metricsets: ["broker","queue","topic"]
              enabled: true
              period: 60s
              hosts: ["${data.host}:8161/api/jolokia"]
              username: ${USERNAME_FROM_DOCKER_SECRETS}
              password: ${PASSWORD_FROM_DOCKER_SECRETS}
              namespace: "my_namespace"

The setup works if I hardcode the credentials in the configuration file. Which obviously is something that I do not want to do.

[1] How to use secrets in Docker Compose | Docker Docs
[2] Secrets keystore for secure settings | Metricbeat Reference [8.11] | Elastic

[ABL]

Hi Denis,
I am currently looking to do something similar and am curious to hear if you have found a suitable solution to your problem?

[strawgate (William Easton)]

I don't believe the modules support loading passwords directly from a file which likely precludes you from just mounting the secret and referencing it directly from the configuration.

That being said it would be relatively trivial to create an entrypoint script that dynamically takes any mounted secrets and adds them to the keystore and then invokes Filebeat.

Docker secrets mount to /run/secrets/ by default I believe so you'd just create a bash script which iterates through the files in that directory and adds it to the keystore.

Something like:

filebeat keystore create
for secret in /run/secrets/*; do
 cat /run/secrets/$secret | filebeat keystore add $secret --stdin --force
done

...invoke filebeat

You can then reference the docker secrets by name in the config password: "${DOCKER_SECRET_ONE}"

I wrote this free hand so usual caveats apply of testing and confirming the behavior and working through any syntax or other issues like escaping or quoting values when using the example provided above.

[denisk (Denis)]

Thanks for the suggestion William, that looks promising.