Metricbeat does not get *_state metrics from kube-state-metrics

Elastic Stack Beats
1

Hi
I've deployed Metricbeat to my OpenShift environment along with kube-state-metrics as it's supposed to be used to get pod and container level metrics (such as CPU, RAM, IO etc). In documentation I read the kubernetes module gets state_pod metricset from kube-state-metrics.
I have default metricbeat-deployment-modules ConfigMap:

apiVersion: v1
data:
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        # Uncomment this to get k8s events:
        #- event
      period: 10s
      hosts: ["kube-state-metrics:8080"]
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    k8s-app: metricbeat
    kubernetes.io/cluster-service: "true"
  name: metricbeat-deployment-modules

In the events of Metricbeat I see events like following:

2018-04-20T08:00:06.581Z        DEBUG   [publish]       pipeline/processor.go:275       Publish event: {
  "@timestamp": "2018-04-20T08:00:06.568Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.2.3"
  },
  "metricset": {
    "rtt": 12430,
    "namespace": "kubernetes.pod",
    "name": "state_pod",
    "module": "kubernetes",
    "host": "kube-state-metrics:8080"
  },
  "kubernetes": {
    "namespace": "default",
    "pod": {
      "name": "docker-registry-17-8zl0p",
      "ip": "10.129.2.171",
      "host_ip": "10.2.3.18",
      "status": {
        "phase": "running",
        "ready": "true",
        "scheduled": "true"
      }
    },
    "node": {
      "name": ""
    }
  },
  "beat": {
    "version": "6.2.3",
    "name": "metricbeat-3335512447-7hq50",
    "hostname": "metricbeat-3335512447-7hq50"
  }
}

I expected to see much more fields in pod information as described here: https://www.elastic.co/guide/en/beats/metricbeat/6.2/exported-fields-kubernetes.html

I checked output of kube-state-metrics:8080/metrics and saw quite much of pod data returned:

# HELP kube_pod_container_requested_memory_bytes The number of requested memory bytes  by a container.
# TYPE kube_pod_container_requested_memory_bytes gauge
kube_pod_container_requested_memory_bytes{container="curator",namespace="logging",node="inf-ph-openshift-node02.lan.local",pod="logging-curator-2-n7vdl"} 2.68435456e+08
kube_pod_container_requested_memory_bytes{container="elasticsearch",namespace="logging",node="inf-ph-openshift-node02.lan.local",pod="logging-es-data-master-zmmlgtku-2-fchgw"} 8.589934592e+09
kube_pod_container_requested_memory_bytes{container="fluentd-elasticsearch",namespace="logging",node="inf-ph-openshift-master01.lan.local",pod="logging-fluentd-q5pd2"} 5.36870912e+08
kube_pod_container_requested_memory_bytes{container="fluentd-elasticsearch",namespace="logging",node="inf-ph-openshift-master02.lan.local",pod="logging-fluentd-vd92p"} 5.36870912e+08
kube_pod_container_requested_memory_bytes{container="fluentd-elasticsearch",namespace="logging",node="inf-ph-openshift-master03.lan.local",pod="logging-fluentd-hdljn"} 5.36870912e+08
kube_pod_container_requested_memory_bytes{container="fluentd-elasticsearch",namespace="logging",node="inf-ph-openshift-node01.lan.local",pod="logging-fluentd-kksp0"} 5.36870912e+08
kube_pod_container_requested_memory_bytes{container="fluentd-elasticsearch",namespace="logging",node="inf-ph-openshift-node02.lan.local",pod="logging-fluentd-nwl6x"} 5.36870912e+08
kube_pod_container_requested_memory_bytes{container="kibana",namespace="logging",node="inf-ph-openshift-node02.lan.local",pod="logging-kibana-2-mnn22"} 7.71751936e+08
kube_pod_container_requested_memory_bytes{container="kibana-proxy",namespace="logging",node="inf-ph-openshift-node02.lan.local",pod="logging-kibana-2-mnn22"} 2.68435456e+08
kube_pod_container_requested_memory_bytes{container="metricbeat",namespace="logging",node="inf-ph-openshift-node01.lan.local",pod="metricbeat-3335512447-7hq50"} 1.048576e+08
kube_pod_container_requested_memory_bytes{container="registry",namespace="default",node="inf-ph-openshift-node02.lan.local",pod="docker-registry-17-8zl0p"} 2.68435456e+08
kube_pod_container_requested_memory_bytes{container="router",namespace="default",node="inf-ph-openshift-node01.lan.local",pod="router-19-v62t7"} 2.68435456e+08
kube_pod_container_requested_memory_bytes{container="router",namespace="default",node="inf-ph-openshift-node02.lan.local",pod="router-19-sbhkw"} 2.68435456e+08

However I'm not sure how this data is supposed to be mapped to fields published by Metricbeat. Should it be configured or it should be mapped and published by default?

2

I've figured out from kubernetes module sources the state_pod isn't supposed to contain any CPU or RAM related metrics. But state_container is supposed to have this data though in my case it still doesn't:

2018-04-20T08:48:31.473Z        DEBUG   [publish]       pipeline/processor.go:275       Publish event: {
  "@timestamp": "2018-04-20T08:48:31.455Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "doc",
    "version": "6.2.3"
  },
  "metricset": {
    "name": "state_container",
    "module": "kubernetes",
    "host": "kube-state-metrics:8080",
    "rtt": 18195,
    "namespace": "kubernetes.container"
  },
  "kubernetes": {
    "pod": {
      "name": "logging-fluentd-q5pd2"
    },
    "namespace": "logging",
    "container": {
      "name": "fluentd-elasticsearch",
      "id": "docker://f687fda10019717fb09d4d468290524d6c6894affad7306a0c6100eb40d5df61",
      "image": "docker-registry.default.svc:5000/logging/openshift-logging-fluentd-metrics:latest",
      "status": {
        "restarts": 0,
        "ready": true,
        "phase": "running"
      }
    }
  },
  "beat": {
    "name": "metricbeat-3335512447-7hq50",
    "hostname": "metricbeat-3335512447-7hq50",
    "version": "6.2.3"
  }
}
3

Hi @ralfeus,

What version of kube-state-metrics are you using? It looks like an old one, Metricbeat expects v0.5 or later

Best regards

4

I didn't find the way to find out kube-state-metrics version.
But I used image gcr.io/google_containers/kube-state-metrics:v0.3.0 if that says anything

5

Yes, that means version is v0.3.0, have a look to https://github.com/kubernetes/kube-state-metrics#kubernetes-deployment to deploy a newer version

6

Yes! This one works fine :slight_smile:
Thanks a lot!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.