Metricbeat kubernetes WARN unable to index event

We only have one warning level error being logged by metricbeat which stated as unable to index an event with a mapper_parsing_exception. Could you please advise on how to correct this.

log sample:
2018-09-03T14:13:05.055Z WARN elasticsearch/client.go:520 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbedb6e3b9d530d18, ext:122658578627, loc:(*time.Location)(0x37d2d20)}, Meta:common.MapStr(nil), Fields:common.MapStr{"beat":common.MapStr{"name":"ip-120-0-12-235", "hostname":"ip-120-0-12-235", "version":"6.4.0"}, "host":common.MapStr{"name":"ip-120-0-12-235"}, "meta":common.MapStr{"cloud":common.MapStr{"instance_id":"i-00c63fa0e261452f6", "machine_type":"c5.xlarge", "region":"us-east-1", "availability_zone":"us-east-1b", "provider":"ec2"}}, "metricset":common.MapStr{"rtt":1806499, "namespace":"kubernetes.container", "name":"state_container", "module":"kubernetes", "host":"kube-state-metrics.monitoring.svc:8080"}, "kubernetes":common.MapStr{"namespace":"env-stg", "pod":common.MapStr{"name":"app-proxy-7d459f4d88-rd7vp"}, "node":common.MapStr{"name":"ip-120-0-12-235.us-east-1.compute.internal"}, "labels":common.MapStr{"role":"proxy", "env":"env-stg", "k8s-app":"app", "pod-template-hash":"3801590844"}, "container":common.MapStr{"id":"docker://38b4dc42ad8cda23b80f364872286016a133a59d69bf3d7f4ecf0462b1ca6739", "name":"nginx", "status":common.MapStr{"phase":"running", "restarts":0, "ready":true}, "image":"accountno.dkr.ecr.us-east-1.amazonaws.com/registry/nginx:latest"}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"object mapping for [kubernetes.labels.role] tried to parse field [role] as object, but found a concrete value"}

Hi Alex_Armstrong,

It looks like role label is used in different ways across your cluster, where some use role.something: value and this pod uses role: value. Sounds like this is the case?

As dots represent nesting in Elasticsearch, this is causing mapping issues (role is expected to be an object as for previous documents). You can use rename processor to rename role to something else that won't break the mapping: https://www.elastic.co/guide/en/beats/metricbeat/current/rename-fields.html

Best regards

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.