I'm trying to collect data from Kubernetes cluster using Metricbeat and Kube-state-metrics. My configuration is the same as https://raw.githubusercontent.com/elastic/beats/7.3/deploy/kubernetes/metricbeat-kubernetes.yaml (can't use 7.4.* because of bugs) and only differs from the official examples by ELASTICSEARCH_HOST and
spec.template.spec.nodeSelector.beta.kubernetes.io/os: linux
In the log for Metricbeat pod, which reads Kube-state-metrics, I see a lot of warnings like:
log example
2019-11-12T16:24:55.676Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf6ad515a562a615, ext:93052019337, loc:(*time.Location)(0x5716aa0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"26e8b145-53d5-4867-a59d-2dc8a19c4d49", "hostname":"cl-lnx1", "id":"95d3636a-0716-48dd-a001-4a90944ebc5e", "type":"metricbeat", "version":"7.3.2"}, "ecs":common.MapStr{"version":"1.0.1"}, "event":common.MapStr{"dataset":"kubernetes.pod", "duration":21352450, "module":"kubernetes"}, "host":common.MapStr{"name":"cl-lnx1"}, "kubernetes":common.MapStr{"labels":common.MapStr{"app":common.MapStr{"kubernetes":common.MapStr{"io/name":"kube-state-metrics", "io/version":"v1.8.0"}}, "pod-template-hash":"5458dddb44"}, "namespace":"kube-system", "node":common.MapStr{"name":"cl-lnx1"}, "pod":common.MapStr{"host_ip":"172.30.200.5", "ip":"10.244.3.3", "name":"kube-state-metrics-5458dddb44-zrzgq", "status":common.MapStr{"phase":"running", "ready":"true", "scheduled":"true"}, "uid":"4a51ecdf-0212-11ea-8d41-005056bfb6c2"}}, "metricset":common.MapStr{"name":"state_pod"}, "service":common.MapStr{"address":"kube-state-metrics:8080", "type":"kubernetes"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'd-5tYG4BzYQ8BvX12mMf'. Preview of field's value: '{kubernetes={io/name=kube-state-metrics, io/version=v1.8.0}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:389"}}
What is wrong? Kube-state-metrics output doesn't match what Metricbeat expects? How do I make them agree? Should I downgrade kube-state-metrics too?