Metricbeat reports an error for unconfigured auto discovery hints in Kubernetes and ECK

We are using hints based auto discovery for metricteat 7.13.0 on ECK 1.6.0, we keep geeting this error:

ERROR [autodiscover] autodiscover/autodiscover.go:212 Auto discover config check failed for config '{
"metricsets": [
"pending_tasks",
"node_stats"
],
"module": "elasticsearch",
"period": "1m",
"timeout": "3s"
}', won't start runner: 4 errors: host parsing failed for elasticsearch-shard: error parsing URL: empty host; host parsing failed for elasticsearch-node: error parsing URL: empty host; host parsing failed for elasticsearch-pending_tasks: error parsing URL: empty host; host parsing failed for elasticsearch-node_stats: error parsing URL: empty host

ERROR [autodiscover] autodiscover/autodiscover.go:212 Auto discover config check failed for config '{
"enabled": true,
"metricsets": [
"node",
"pending_tasks",
"shard",
"node_stats"
],

We are using the following configurations and I also want to put into consideration that cluster monitoring is working for all components.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: elastic-mb
  namespace: elasticsearch
spec:
  type: metricbeat
  version: 7.13.0
  elasticsearchRef:
    name: elastic
  kibanaRef:
    name: elastic
  config:
    logging.json: true
    logging.level: warning
    monitoring.enabled: true
    metricbeat:
      autodiscover:
        providers:
        - type: kubernetes
          scope: cluster
          hints.enabled: true
          templates:
            - condition:
                contains:
                  kubernetes.labels.elastic-name: elastic
              config:
                - module: elasticsearch
                  metricsets:
                    - ccr
                    - cluster_stats
                    - enrich
                    - index
                    - index_recovery
                    - index_summary
                    - ml_job
                    - node_stats
                    - shard
                  period: 10s
                  hosts: "https://${data.host}:${data.ports.https}"
                  username: ${MONITORED_ES_USERNAME}
                  password: ${MONITORED_ES_PASSWORD}
                  ssl.certificate_authorities:
                    - /mnt/elastic-internal/elasticsearch-certs/ca.crt
                  xpack.enabled: true
            - condition:
                contains:
                  kubernetes.labels.kibana-name: elastic
              config:
                - module: kibana
                  metricsets:
                    - stats
                  period: 10s
                  hosts: "https://${data.host}:${data.ports.https}"
                  username: ${MONITORED_ES_USERNAME}
                  password: ${MONITORED_ES_PASSWORD}
                  ssl.certificate_authorities:
                    - /mnt/elastic-internal/kibana-certs/ca.crt
                  xpack.enabled: true
        - type: kubernetes
          scope: cluster
          node: ${NODE_NAME}
          unique: true
          templates:
            - config:
                - module: kubernetes
                  hosts: [ "kube-state-metrics.kube-system:8080" ]
                  period: 10s
                  add_metadata: true
                  metricsets:
                    - state_node
                    - state_deployment
                    - state_daemonset
                    - state_replicaset
                    - state_pod
                    - state_container
                    - state_cronjob
                    - state_resourcequota
                    - state_statefulset
                    - state_service
                - module: kubernetes
                  metricsets:
                    - apiserver
                  hosts: [ "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}" ]
                  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
                  ssl.certificate_authorities:
                    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
                  period: 30s
      modules:
      - module: system
        period: 10s
        metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        processes: [ '.*' ]
        process.include_top_n:
          by_cpu: 5      # include top 5 processes by CPU
          by_memory: 5   # include top 5 processes by memory
      - module: system
        period: 1m
        metricsets:
        - filesystem
        - fsstat
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
      - module: kubernetes
        metricsets:
        - node
        - system
        - pod
        - container
        - volume
        period: 10s
        host: ${NODE_NAME}
        hosts: [ "https://${NODE_NAME}:10250" ]
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl.certificate_authorities:
          - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      - module: kubernetes
        metricsets:
        - proxy
        period: 10s
        host: ${NODE_NAME}
        hosts: [ "localhost:10249" ]
    processors:
    - add_cloud_metadata: {}
    - add_host_metadata: {}
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: metricbeat
        automountServiceAccountToken: true
        terminationGracePeriodSeconds: 30
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        containers:
        - name: metricbeat
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 200Mi
          args: [
            "-c", "/etc/beat.yml",
            "-e",
            "-system.hostfs=/hostfs",
          ]
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: MONITORED_ES_USERNAME
            value: elastic
          - name: MONITORED_ES_PASSWORD
            valueFrom:
              secretKeyRef:
                key: elastic
                name: elastic-es-elastic-user
          securityContext:
            runAsUser: 0
          volumeMounts:
          - name: data
            mountPath: /usr/share/metricbeat/data
          - name: proc
            mountPath: /hostfs/proc
            readOnly: true
          - name: cgroup
            mountPath: /hostfs/sys/fs/cgroup
            readOnly: true
        volumes:
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: data
          hostPath:
            path: /var/lib/metricbeat-data
            type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  - secrets
  - services
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- apiGroups:
    - coordination.k8s.io
  resources:
    - leases
  verbs:
    - '*'
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: elasticsearch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: elasticsearch
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io

Please also note that the autodiscovery rule configured is working, and it is diffrent from what is reporting error, u can notice the period of 1m compared to 10s configured in the yaml and the metricsets also.
Can you help me to identify why this is happening and how to solve.
Thank you very much

Hi Elastic team;
Did anyone have the change to look into this?
Thanks in advance.

Try making the hosts field for both the elasticsearch and kibana modules arrays, by enclosing the hosts value with brackets [].

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.