Hello,
I am using:
- ELK 7.4.2
- quay.io/coreos/kube-state-metrics:v1.8.0
- Metricbeat 7.4.2
Metricbeat-kubernetes.yaml
---
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: monitoring
labels:
k8s-app: metricbeat
data:
# This module requires `kube-state-metrics` up and running under `kube-system` namespace
kubernetes.yml: |-
- module: kubernetes
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_statefulset
- state_pod
- state_container
#- state_cronjob
# Uncomment this to get k8s events:
#- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics.monitoring:8080"]
---
# Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: monitoring
labels:
k8s-app: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat:7.4.2
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: ******
- name: ELASTICSEARCH_PORT
value: "9200"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-deployment-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: monitoring
roleRef:
kind: ClusterRole
name: metricbeat-admin
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
- nodes/metrics
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: monitoring
labels:
k8s-app: metricbeat
---
In app kube-state-metrics I see metrics:
kube_pod_container_resource_limits_cpu_cores{namespace="***",pod="***-gkmg8",container="speaker",node="i-l-worker-1"} 0.1
kube_pod_container_resource_requests_cpu_cores{namespace="***",pod="***-7679b6cd45-hmvzk",container="***",node="i-l-worker-1"} 0.25
kube_pod_container_resource_requests_memory_bytes{namespace="**",pod="**-747f79f787-z69gf",container="**",node="i-l-worker-2"} 5.36870912e+08
kube_pod_container_resource_requests_memory_bytes{namespace="**",pod="**-d77454c4d-pzfw4",container="**",node="i-l-worker-2"} 2.147483648e+09
# HELP kube_pod_container_resource_limits_memory_bytes The limit on memory to be used by a container in bytes.
# TYPE kube_pod_container_resource_limits_memory_bytes gauge
kube_pod_container_resource_limits_memory_bytes{namespace="**",pod="**-5c98db65d4-4s6wv",container="**",node="i-l-k8s-1"} 1.7825792e+08
v
In ELK no metrics cpu, memory for kubernetes pods:
JSON document metricset=state_pod
{
"_index": "metricbeat-7.4.2-2019.11.12-000001",
"_type": "_doc",
"_id": "HBJpX24BQulC12ySEAtR",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2019-11-12T11:39:10.084Z",
"metricset": {
"period": 10000,
"name": "state_pod"
},
"service": {
"address": "kube-state-metrics.monitoring:8080",
"type": "kubernetes"
},
"event": {
"module": "kubernetes",
"duration": 79482011,
"dataset": "kubernetes.pod"
},
"kubernetes": {
"namespace": "***",
"node": {
"name": "i-l-worker-3"
},
"labels": {
"app": "***",
"pod-template-hash": "54bbc6d565"
},
"replicaset": {
"name": "***-54bbc6d565"
},
"pod": {
"name": "***-54bbc6d565-6r828",
"status": {
"ready": "true",
"scheduled": "true",
"phase": "running"
},
"host_ip": "172.16.15.78",
"ip": "10.35.0.13",
"uid": "79446894-b97a-4072-97d5-b08da0d7ae9e"
}
},
"ecs": {
"version": "1.1.0"
},
"host": {
"name": "i-l-worker-2"
},
"agent": {
"id": "a2d2772e-4dbb-4c6e-b0ea-eafdbdd13698",
"version": "7.4.2",
"type": "metricbeat",
"ephemeral_id": "e83bbccb-ef8b-4f69-84f4-d44c62e07802",
"hostname": "i-l-worker-2"
}
},
"fields": {
"@timestamp": [
"2019-11-12T11:39:10.084Z"
]
},
"highlight": {
"metricset.name": [
"@kibana-highlighted-field@state_pod@/kibana-highlighted-field@"
]
},
"sort": [
1573558750084
]
}