Hi all, I deployed metricbeat under Kubernetes and am getting warnings that I interpret to mean that Elasticsearch is unhappy with pod label attributes being sent for kube-state-metrics. More specifically, the "standard" deployment of kube-state-metrics labels its pods with app.kubernetes.io/name: kube-state-metrics and app.kubernetes.io/version: 1.9.5. But the dynamic_templates that metricbeats create declare that kubernetes.labels.* are keywords, and (AFAIK) can only accept string values for app, not the deeper structure being sent.
Here are some excerpts from the warning to clarify:
2020-03-06T22:17:24.947Z WARN elasticsearch/client.go:517 Cannot index event publisher.Event{
...
"kubernetes":{"labels":{"app":{"kubernetes":{"io/name":"kube-state-metrics","io/version":"1.9.5"}}
...
(status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id '6vLssXABJgDINP2ltFLH'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:570"}}
Anybody know if there are ways to cope with this in the metricbeat config? (Or do I have to change the standard labeling of this pod)?
Thanks,
-Joe