No metrics cpu, memory for kubernetes pods

Hello,
I am using:

  1. ELK 7.4.2
  2. quay.io/coreos/kube-state-metrics:v1.8.0
  3. Metricbeat 7.4.2
Metricbeat-kubernetes.yaml
 ---
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: metricbeat-deployment-modules
    namespace: monitoring
    labels:
      k8s-app: metricbeat
  data:
    # This module requires `kube-state-metrics` up and running under `kube-system` namespace
    kubernetes.yml: |-
      - module: kubernetes
        metricsets:
          - state_node
          - state_deployment
          - state_replicaset
          - state_statefulset
          - state_pod
          - state_container
          #- state_cronjob
          # Uncomment this to get k8s events:
          #- event
        period: 10s
        host: ${NODE_NAME}
        hosts: ["kube-state-metrics.monitoring:8080"]
  ---
  # Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
  apiVersion: apps/v1beta1
  kind: Deployment
  metadata:
    name: metricbeat
    namespace: monitoring
    labels:
      k8s-app: metricbeat
  spec:
    template:
      metadata:
        labels:
          k8s-app: metricbeat
      spec:
        serviceAccountName: metricbeat
        hostNetwork: true
        dnsPolicy: ClusterFirstWithHostNet
        containers:
        - name: metricbeat
          image: docker.elastic.co/beats/metricbeat:7.4.2
          args: [
            "-c", "/etc/metricbeat.yml",
            "-e",
          ]
          env:
          - name: ELASTICSEARCH_HOST
            value: ******
          - name: ELASTICSEARCH_PORT
            value: "9200"
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
          volumeMounts:
          - name: config
            mountPath: /etc/metricbeat.yml
            readOnly: true
            subPath: metricbeat.yml
          - name: modules
            mountPath: /usr/share/metricbeat/modules.d
            readOnly: true
        volumes:
        - name: config
          configMap:
            defaultMode: 0600
            name: metricbeat-deployment-config
        - name: modules
          configMap:
            defaultMode: 0600
            name: metricbeat-deployment-modules
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRoleBinding
  metadata:
    name: metricbeat
  subjects:
  - kind: ServiceAccount
    name: metricbeat
    namespace: monitoring
  roleRef:
    kind: ClusterRole
    name: metricbeat-admin
    apiGroup: rbac.authorization.k8s.io
  ---
  apiVersion: rbac.authorization.k8s.io/v1beta1
  kind: ClusterRole
  metadata:
    name: metricbeat
    labels:
      k8s-app: metricbeat
  rules:
  - apiGroups: [""]
    resources:
    - nodes
    - namespaces
    - events
    - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
    - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
    - statefulsets
    - deployments
    verbs: ["get", "list", "watch"]
  - apiGroups:
    - ""
    resources:
    - nodes/stats
    - nodes/metrics
    verbs:
    - get
  ---
  apiVersion: v1
  kind: ServiceAccount
  metadata:
    name: metricbeat
    namespace: monitoring
    labels:
      k8s-app: metricbeat
  ---
In app kube-state-metrics I see metrics:
  kube_pod_container_resource_limits_cpu_cores{namespace="***",pod="***-gkmg8",container="speaker",node="i-l-worker-1"} 0.1
  kube_pod_container_resource_requests_cpu_cores{namespace="***",pod="***-7679b6cd45-hmvzk",container="***",node="i-l-worker-1"} 0.25
  kube_pod_container_resource_requests_memory_bytes{namespace="**",pod="**-747f79f787-z69gf",container="**",node="i-l-worker-2"} 5.36870912e+08
  kube_pod_container_resource_requests_memory_bytes{namespace="**",pod="**-d77454c4d-pzfw4",container="**",node="i-l-worker-2"} 2.147483648e+09
  # HELP kube_pod_container_resource_limits_memory_bytes The limit on memory to be used by a container in bytes.
  # TYPE kube_pod_container_resource_limits_memory_bytes gauge
  kube_pod_container_resource_limits_memory_bytes{namespace="**",pod="**-5c98db65d4-4s6wv",container="**",node="i-l-k8s-1"} 1.7825792e+08

v

In ELK no metrics cpu, memory for kubernetes pods:

JSON document metricset=state_pod
{
"_index": "metricbeat-7.4.2-2019.11.12-000001",
"_type": "_doc",
"_id": "HBJpX24BQulC12ySEAtR",
"_version": 1,
"_score": null,
"_source": {
  "@timestamp": "2019-11-12T11:39:10.084Z",
  "metricset": {
    "period": 10000,
    "name": "state_pod"
  },
  "service": {
    "address": "kube-state-metrics.monitoring:8080",
    "type": "kubernetes"
  },
  "event": {
    "module": "kubernetes",
    "duration": 79482011,
    "dataset": "kubernetes.pod"
  },
  "kubernetes": {
    "namespace": "***",
    "node": {
      "name": "i-l-worker-3"
    },
    "labels": {
      "app": "***",
      "pod-template-hash": "54bbc6d565"
    },
    "replicaset": {
      "name": "***-54bbc6d565"
    },
    "pod": {
      "name": "***-54bbc6d565-6r828",
      "status": {
        "ready": "true",
        "scheduled": "true",
        "phase": "running"
      },
      "host_ip": "172.16.15.78",
      "ip": "10.35.0.13",
      "uid": "79446894-b97a-4072-97d5-b08da0d7ae9e"
    }
  },
  "ecs": {
    "version": "1.1.0"
  },
  "host": {
    "name": "i-l-worker-2"
  },
  "agent": {
    "id": "a2d2772e-4dbb-4c6e-b0ea-eafdbdd13698",
    "version": "7.4.2",
    "type": "metricbeat",
    "ephemeral_id": "e83bbccb-ef8b-4f69-84f4-d44c62e07802",
    "hostname": "i-l-worker-2"
  }
},
"fields": {
  "@timestamp": [
    "2019-11-12T11:39:10.084Z"
  ]
},
"highlight": {
  "metricset.name": [
    "@kibana-highlighted-field@state_pod@/kibana-highlighted-field@"
  ]
},
"sort": [
  1573558750084
]

}

See Metricbeats on GKE not working as expected - Missing deployments, nodes, etc and https://github.com/elastic/beats/pull/13905

I downgraded Metricbeat to 7.3.2 and this allowed me to see correct graphs. I still can't get CPU, memory and network.

1 Like

Did you try to execute the commands as an admin? I did it and It worked.