Metricbeat Deployment: decoding of metric family failed

[JonasDeGendt]

Trying to run Metricbeat 7.4 on Kubernetes, my Daemonset metricbeat is running fine, but the singleton Deployment type returns the following error message:

2019-11-05T14:20:31.748Z        ERROR   [kubernetes.state_statefulset]  state_statefulset/state_statefulset.go:97       decoding of metric family failed: text format parsing error in line 1: expected float as value, got "(user=system:serviceaccount:monitoring:metricbeat,"

Any insights welcome! Adding my deployment configmap below as well.

apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: monitoring
  labels:
    k8s-app: metricbeat
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      period: 10s
      hosts: ["localhost:10255"] #kubelet
      enabled: true
      add_metadata: true

    - module: kubernetes
      enabled: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
      period: 10s
      hosts: ["https://kube-state-metrics.monitoring.svc.cluster.local:8443"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
       - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      ssl.verification_mode: none

    - module: kubernetes
      enabled: true
      metricsets:
        - apiserver
      hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"] #apiserver

    - module: kubernetes
      enabled: true
      metricsets:
        - controllermanager
      hosts: ["http://localhost:10252"] #controllermanager
      period: 10s

    - module: kubernetes
      enabled: true
      metricsets:
        - proxy
      hosts: ["http://localhost:10249"] #apiserver
      period: 10s

    - module: kubernetes
      enabled: true
      metricsets:
        - scheduler
      hosts: ["http://localhost:10251"] #scheduler
      period: 10s

The document in Elasticsearch implies this is related to kube-state-metrics?

{
  "_index": "metricbeat-sli-aksm-weu-01-2019.45",
  "_type": "_doc",
  "_id": "BKAcPG4B7RuRUpcd-8u_",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2019-11-05T15:09:22.047Z",
    "service": {
      "address": "kube-state-metrics.monitoring.svc.cluster.local:8443",
      "type": "kubernetes"
    },
    "error": {
      "message": "decoding of metric family failed: text format parsing error in line 1: expected float as value, got \"(user=system:serviceaccount:monitoring:metricbeat,\""
    },
    "event": {
      "dataset": "kubernetes.state_deployment",
      "module": "kubernetes",
      "duration": 1063001
    },
    "ecs": {
      "version": "1.1.0"
    },
    "cloud": {
      "provider": "az",
      "instance": {
        "id": "redacted",
        "name": "redacted"
      },
      "machine": {
        "type": "Standard_F16s"
      },
      "region": "westeurope"
    },
    "metricset": {
      "name": "state_deployment",
      "period": 10000
    },
    "tags": [
      "sli-aksm-weu-01",
      "westeurope"
    ],
    "host": {
      "hostname": "aks-pool01-31470587-2",
      "architecture": "x86_64",
      "os": {
        "name": "CentOS Linux",
        "kernel": "4.15.0-1061-azure",
        "codename": "Core",
        "platform": "centos",
        "version": "7 (Core)",
        "family": "redhat"
      },
      "containerized": false,
      "name": "aks-pool01-31470587-2"
    },
    "agent": {
      "id": "e8a7bda8-9469-4cbf-9ac2-f9e11a5fbeb2",
      "version": "7.4.2",
      "type": "metricbeat",
      "ephemeral_id": "79cbfa5e-821e-4c8f-8fbb-4ac2fdb64146",
      "hostname": "aks-pool01-31470587-2"
    }
  },
  "fields": {
    "@timestamp": [
      "2019-11-05T15:09:22.047Z"
    ]
  },
  "sort": [
    1572966562047
  ]
}