Metricbeat kubernetes module state_node metricset failed to parse node labels

[Jun_Zhang]

metricbeat log：

2018-11-12T17:54:45.479+0800 WARN elasticsearch/client.go:520 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbef27156c82fe362, ext:1463147003462, loc:(*time.Location)(0x37e0b20)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"host":"172.27.128.71:8794", "rtt":9019564, "namespace":"kubernetes.
node", "name":"state_node", "module":"kubernetes"}, "kubernetes":common.MapStr{"labels":common.MapStr{"ip-masq-fix":"true", "node":common.MapStr{"alpha":common.MapStr{"4pd":common.MapStr{"io/group-group1112":"true"}}}, "beta":common.MapStr{"kubernetes":common.MapStr{"io/os":"linux", "io/arch":"amd64"}}, "kubernetes":common.MapStr{"io/hostname":"m7-devops-128054"
}, "prophet":common.MapStr{"4paradigm":common.MapStr{"com/offline":"true", "com/app":"true", "com/elasticsearch":"true", "com/online":"true", "com/system":"true", "com/addon":"true"}, "value":"true"}, "es-data":"true"}, "node":common.MapStr{"pod":common.MapStr{"capacity":common.MapStr{"total":220}, "allocatable":common.MapStr{"total":220}}, "status":common.MapSt
r{"unschedulable":false, "ready":"true"}, "memory":common.MapStr{"allocatable":common.MapStr{"bytes":2.70272487424e+11}, "capacity":common.MapStr{"bytes":2.70377345024e+11}}, "cpu":common.MapStr{"allocatable":common.MapStr{"cores":40}, "capacity":common.MapStr{"cores":40}}, "name":"m7-devops-128054"}}, "beat":common.MapStr{"name":"metricbeat-128071-bin", "hostna
me":"m7-devops-128071", "version":"6.4.3"}, "host":common.MapStr{"name":"m7-devops-128071", "id":"61432cc588f747f4aec71029ea9e9408", "containerized":true, "architecture":"x86_64", "os":common.MapStr{"platform":"centos", "version":"7 (Core)", "family":"redhat", "codename":"Core"}}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"mapper_parsing_exce
ption","reason":"failed to parse [kubernetes.labels.node]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:222"}}

kubectl get node m7-devops-128054 --show-labels

NAME STATUS ROLES AGE VERSION LABELS
m7-devops-128054 Ready 67d v1.8.15 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,es-data=true,ip-masq-fix=true,kubernetes.io/hostname=m7-devops-128054,node.alpha.4pd.io/group-group1112=true,prophet.4paradigm.com/addon=true,prophet.4paradigm.com/app=true,prophet.4paradigm.com/elasticsearch=true,prophet.4paradigm.com/offline=true,prophet.4paradigm.com/online=true,prophet.4paradigm.com/system=true,prophet=true

[jsoriano (Jaime Soriano)]

Hi @Jun_Zhang,

It seems that elasticsearch is not being able to parse kubernetes.labels.node according to the types it has in the index mapping. This can be caused by having fields with different types in different events, e.g if kubernetes.labels.node is a string in some events and an object containing more fields in other events.

To confirm that this is the problem you can try to drop the conflicting fields adding a drop_field processor to filebeat, something like:

processors:
 - drop_fields:
     fields:
       - "kubernetes.labels.node"

Then you should be able to see events, but without these labels.

Of course this is not optimal because these labels are lost, we are evaluating possible solutions to these conflicts for future versions.

In the meantime, as a workaround, you can try to use the rename processor to change the field name of labels you are interested, and drop_fields to drop the conflicting ones.

[Jun_Zhang]

Got. Thanks!