Does metricbeat hint based autodiscovery support kubernetes service resources?

[JorenThijs (Joren)]

I am trying to configure metricbeat using autodiscovery for monitoring our RabbitMQ cluster.
Our rabbitMQ cluster has 3 nodes. Putting autodiscovery hints on the pods makes our logs duplicated 3 times. See Configure Metricbeat for RabbitMQ cluster to avoid duplication per node - Elastic Stack / Beats - Discuss the Elastic Stack.

I instead want to apply the autodiscover hints to the service that bundles these nodes but hint based autodiscovery doesnt seem to work, and adding them also doen't trigger any behaviour or logs in the debug loging output from metricbeat.

Therefore I want to ask wheter this is supported?

I configured our metricbeat autodicover provided to look for service resources. And enabled hint based autodiscovery.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: cluster-monitoring-mb
spec:
  type: metricbeat
  version: 7.17.18
  config:
    metricbeat:
      autodiscover:
        providers:
          - type: kubernetes # leader (cluster) config
            scope: cluster
            unique: true
            resource: service
            hints:
              enabled: true
            templates:
              - config:
                  - module: kubernetes #this is for collecting kubernetes events from the cluster
                    metricsets:
                      - event

configuring pod annotations: (this works)

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbit-mq
spec:
  image: masstransit/rabbitmq:3.9
  replicas: 3 # Must be an odd number, see https://www.rabbitmq.com/clustering.html#node-count
  override:
    statefulSet:
      spec:
        template:
          metadata:
            annotations:
              co.elastic.metrics/module: rabbitmq
              co.elastic.metrics/hosts: "${data.host}:15672/rmq-mgmt"
              co.elastic.metrics/metricsets: "node, queue"
              co.elastic.metrics/period: 60s
              co.elastic.metrics/username: ${kubernetes.${K8S_NAMESPACE}$.rabbit-mq-default-user.username}
              co.elastic.metrics/password: ${kubernetes.${K8S_NAMESPACE}$.rabbit-mq-default-user.password}

configuring service annotations: (this does not work)

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbit-mq
spec:
  resources:
    requests:
      cpu: 1
      memory: 4Gi
    limits:
      cpu: 2
      memory: 4Gi
  image: masstransit/rabbitmq:3.11 # We pin the minor version to avoid feature-flags conflicts between nodes in a cluster. See https://www.rabbitmq.com/feature-flags.html
  replicas: 3 # Must be an odd number, see https://www.rabbitmq.com/clustering.html#node-count
  override:
    service:
      metadata:
        annotations:
          co.elastic.metrics/module: rabbitmq
          co.elastic.metrics/hosts: "${kubernetes.service.name}:15672/rmq-mgmt"
          co.elastic.metrics/metricsets: "node, queue"
          co.elastic.metrics/period: 60s
          co.elastic.metrics/username: ${kubernetes.${K8S_NAMESPACE}$.rabbit-mq-default-user.username}
          co.elastic.metrics/password: ${kubernetes.${K8S_NAMESPACE}$.rabbit-mq-default-user.password}

When i query the service I can see the annotations being applied.

kubectl describe services/rabbit-mq -n product-dev
Name:              rabbit-mq
Namespace:         product-dev
Labels:            app.kubernetes.io/component=rabbitmq
                   app.kubernetes.io/name=rabbit-mq
                   app.kubernetes.io/part-of=rabbitmq
Annotations:       co.elastic.metrics/hosts: ${data.host}:15672/rmq-mgmt
                   co.elastic.metrics/metricsets: node, queue, exchange
                   co.elastic.metrics/module: rabbitmq
                   co.elastic.metrics/password: ${kubernetes.kaspar123-dev.rabbit-mq-default-user.password}
                   co.elastic.metrics/period: 60s
                   co.elastic.metrics/username: ${kubernetes.kaspar123-dev.rabbit-mq-default-user.username}
                   rabbitmq.com/queueRebalanceNeededAt: 2024-03-13T13:03:03Z
Selector:          app.kubernetes.io/name=rabbit-mq