Cannot index event publisher.Event ... Can't get text on a START_OBJECT

[tkzv]

I'm trying to collect data from Kubernetes cluster using Metricbeat and Kube-state-metrics. My configuration is the same as https://raw.githubusercontent.com/elastic/beats/7.3/deploy/kubernetes/metricbeat-kubernetes.yaml (can't use 7.4.* because of bugs) and only differs from the official examples by ELASTICSEARCH_HOST and
spec.template.spec.nodeSelector.beta.kubernetes.io/os: linux

In the log for Metricbeat pod, which reads Kube-state-metrics, I see a lot of warnings like:

log example

2019-11-12T16:24:55.676Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf6ad515a562a615, ext:93052019337, loc:(*time.Location)(0x5716aa0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"26e8b145-53d5-4867-a59d-2dc8a19c4d49", "hostname":"cl-lnx1", "id":"95d3636a-0716-48dd-a001-4a90944ebc5e", "type":"metricbeat", "version":"7.3.2"}, "ecs":common.MapStr{"version":"1.0.1"}, "event":common.MapStr{"dataset":"kubernetes.pod", "duration":21352450, "module":"kubernetes"}, "host":common.MapStr{"name":"cl-lnx1"}, "kubernetes":common.MapStr{"labels":common.MapStr{"app":common.MapStr{"kubernetes":common.MapStr{"io/name":"kube-state-metrics", "io/version":"v1.8.0"}}, "pod-template-hash":"5458dddb44"}, "namespace":"kube-system", "node":common.MapStr{"name":"cl-lnx1"}, "pod":common.MapStr{"host_ip":"172.30.200.5", "ip":"10.244.3.3", "name":"kube-state-metrics-5458dddb44-zrzgq", "status":common.MapStr{"phase":"running", "ready":"true", "scheduled":"true"}, "uid":"4a51ecdf-0212-11ea-8d41-005056bfb6c2"}}, "metricset":common.MapStr{"name":"state_pod"}, "service":common.MapStr{"address":"kube-state-metrics:8080", "type":"kubernetes"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'd-5tYG4BzYQ8BvX12mMf'. Preview of field's value: '{kubernetes={io/name=kube-state-metrics, io/version=v1.8.0}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:389"}}

What is wrong? Kube-state-metrics output doesn't match what Metricbeat expects? How do I make them agree? Should I downgrade kube-state-metrics too?

[exekias (Carlos Pérez Aradros)]

Hi @tkzv,

It seems labels are causing some issues. Did you disable dabel dedoting?

Best regards

[tkzv]

dabel dedoting

Do you mean label dedotting? I think, it's off by default. Where do I enable it?

I keep getting errors like
ERROR [reload] cfgfile/list.go:96 Error creating runner from config: 1 error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

after I add add_docker_metadata.labels.dedot: true to various processor sections.

Update: I no longer see warnings, other than those socket errors. Metricbeat doesn't seem to get most of data from kube-state-metrics. What does dedot break to make it unable to connect?

Update 2: /var/run/docker.sock is absent from the running container.

[tkzv]

I got no answer to my questions, but I did find a way to prevent this error -- use the label kubernetes.io/os=linux instead of beta.kubernetes.io/os=linux:

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/os: linux

This even allowed to update Metricbeat to 7.4.2. Unfortunately, this does not solve all problems. Some default Kibana visualizations still don't work.