Cannot index event publisher.Event ... Can't get text on a START_OBJECT

I'm trying to collect data from Kubernetes cluster using Metricbeat and Kube-state-metrics. My configuration is the same as https://raw.githubusercontent.com/elastic/beats/7.3/deploy/kubernetes/metricbeat-kubernetes.yaml (can't use 7.4.* because of bugs) and only differs from the official examples by ELASTICSEARCH_HOST and
spec.template.spec.nodeSelector.beta.kubernetes.io/os: linux

In the log for Metricbeat pod, which reads Kube-state-metrics, I see a lot of warnings like:

log example

2019-11-12T16:24:55.676Z WARN elasticsearch/client.go:535 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf6ad515a562a615, ext:93052019337, loc:(*time.Location)(0x5716aa0)}, Meta:common.MapStr(nil), Fields:common.MapStr{"agent":common.MapStr{"ephemeral_id":"26e8b145-53d5-4867-a59d-2dc8a19c4d49", "hostname":"cl-lnx1", "id":"95d3636a-0716-48dd-a001-4a90944ebc5e", "type":"metricbeat", "version":"7.3.2"}, "ecs":common.MapStr{"version":"1.0.1"}, "event":common.MapStr{"dataset":"kubernetes.pod", "duration":21352450, "module":"kubernetes"}, "host":common.MapStr{"name":"cl-lnx1"}, "kubernetes":common.MapStr{"labels":common.MapStr{"app":common.MapStr{"kubernetes":common.MapStr{"io/name":"kube-state-metrics", "io/version":"v1.8.0"}}, "pod-template-hash":"5458dddb44"}, "namespace":"kube-system", "node":common.MapStr{"name":"cl-lnx1"}, "pod":common.MapStr{"host_ip":"172.30.200.5", "ip":"10.244.3.3", "name":"kube-state-metrics-5458dddb44-zrzgq", "status":common.MapStr{"phase":"running", "ready":"true", "scheduled":"true"}, "uid":"4a51ecdf-0212-11ea-8d41-005056bfb6c2"}}, "metricset":common.MapStr{"name":"state_pod"}, "service":common.MapStr{"address":"kube-state-metrics:8080", "type":"kubernetes"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'd-5tYG4BzYQ8BvX12mMf'. Preview of field's value: '{kubernetes={io/name=kube-state-metrics, io/version=v1.8.0}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:389"}}

What is wrong? Kube-state-metrics output doesn't match what Metricbeat expects? How do I make them agree? Should I downgrade kube-state-metrics too?

Hi @tkzv,

It seems labels are causing some issues. Did you disable dabel dedoting?

Best regards

dabel dedoting

Do you mean label dedotting? I think, it's off by default. Where do I enable it?

I keep getting errors like
ERROR [reload] cfgfile/list.go:96 Error creating runner from config: 1 error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

after I add add_docker_metadata.labels.dedot: true to various processor sections.

Update: I no longer see warnings, other than those socket errors. Metricbeat doesn't seem to get most of data from kube-state-metrics. What does dedot break to make it unable to connect?

Update 2: /var/run/docker.sock is absent from the running container.

I got no answer to my questions, but I did find a way to prevent this error -- use the label kubernetes.io/os=linux instead of beta.kubernetes.io/os=linux:

spec:
  template:
    spec:
      nodeSelector:
        kubernetes.io/os: linux

This even allowed to update Metricbeat to 7.4.2. Unfortunately, this does not solve all problems. Some default Kibana visualizations still don't work.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.