Hi all
Apologies if I don't have all the correct terminology as I am new to the elastic stack.
I have just been allocated a task to remedy this problem;
We have a Filebeat - ES - Kibana installation
When users run the following:
https://kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(),filters:!(),index:bc055660-c693-11ea-bcb9-5b73a837b97b,interval:auto,query:(language:kuery,query:'%22Cannot%20index%20event%20publisher.Event%22'),sort:!(!('@timestamp',desc)))
Apparently we used to get results back that included a documents/indexes like:
Expected error for UDA value description. Setting to default value.
now we get back this (which includes the message above):
{
"_index": "filebeat-7.13.3-2022.06.19-000407",
"_type": "_doc",
"_id": "V1XQhoEB-VCaPvu79D9z",
"_version": 1,
"_score": null,
"fields": {
"kubernetes.labels.beat_k8s_elastic_co/version": [
"7.13.3"
],
"kubernetes.node.uid": [
"6f063c79-a3af-4a83-ba49-36dd0cdf90c5"
],
"kubernetes.node.labels.app": [
"hpa"
],
"kubernetes.namespace_uid": [
"4bf344df-36ff-489d-83de-95bf03348ac9"
],
"kubernetes.labels.beat_k8s_elastic_co/config-checksum": [
"2fca66b447485134295d2ed965f1e5e3cd244f384e25bccab18c21b6"
],
"host.os.name.text": [
"CentOS Linux"
],
"kubernetes.node.labels.topology_kubernetes_io/zone": [
"eu-west-1b"
],
"host.hostname": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"host.mac": [
"06:90:e6:51:49:b9",
"06:df:e1:25:7e:a1",
"e2:4f:0c:a7:da:ed",
"7e:6e:5b:af:58:35",
"36:ef:20:0a:ec:9e"
],
"kubernetes.node.labels.kubernetes_io/os": [
"linux"
],
"container.id": [
"7c2cea4c00f9836b405118e8f8166d0320f9abae3004c77c67192626e0375e39"
],
"cloud.availability_zone": [
"eu-west-1b"
],
"kubernetes.node.labels.k8s_amazonaws_com/eniConfig": [
"eu-west-1b-eni"
],
"container.image.name": [
"docker.elastic.co/beats/filebeat:7.13.3"
],
"host.os.version": [
"7 (Core)"
],
"kubernetes.node.labels.vpc_amazonaws_com/eniConfig": [
"eu-west-1b-eni"
],
"kubernetes.node.labels.beta_kubernetes_io/os": [
"linux"
],
"kubernetes.namespace_labels.kustomize_toolkit_fluxcd_io/name": [
"flux-system"
],
"kubernetes.namespace": [
"devops"
],
"host.os.name": [
"CentOS Linux"
],
"agent.name": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"host.name": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"kubernetes.node.labels.topology_kubernetes_io/region": [
"eu-west-1"
],
"host.os.type": [
"linux"
],
"cloud.region": [
"eu-west-1"
],
"kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone": [
"eu-west-1b"
],
"input.type": [
"container"
],
"log.offset": [
6027077
],
"agent.hostname": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"kubernetes.namespace_labels.fluxcd_io/sync-gc-mark": [
"sha256.VJgxYJtEwMtOG1t8L1iOzJ1Ci-nKIEYkx21huD372xM"
],
"host.architecture": [
"x86_64"
],
"cloud.machine.type": [
"c5a.xlarge"
],
"cloud.provider": [
"aws"
],
"container.runtime": [
"docker"
],
"agent.id": [
"17af5615-6f68-43b2-9837-e8d39eee7768"
],
"cloud.service.name": [
"EC2"
],
"ecs.version": [
"1.8.0"
],
"host.containerized": [
true
],
"kubernetes.node.labels.beta_kubernetes_io/instance-type": [
"c5a.xlarge"
],
"agent.version": [
"7.13.3"
],
"host.os.family": [
"redhat"
],
"kubernetes.node.name": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"kubernetes.node.labels.failure-domain_beta_kubernetes_io/region": [
"eu-west-1"
],
"kubernetes.node.hostname": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"kubernetes.node.labels.node_kubernetes_io/instance-type": [
"c5a.xlarge"
],
"kubernetes.pod.uid": [
"61dc703a-d35c-493a-ba50-aac93aa18b48"
],
"host.ip": [
"n.n.n.n",
"fe80::490:e6ff:fe51:49b9",
"100.64.16.38",
"fe80::4df:e1ff:fe25:7ea1",
"fe80::e04f:cff:fea7:daed",
"fe80::7c6e:5bff:feaf:5835",
"fe80::34ef:20ff:fe0a:ec9e"
],
"cloud.instance.id": [
"i-redacted"
],
"agent.type": [
"filebeat"
],
"stream": [
"stderr"
],
"host.os.kernel": [
"5.4.181-99.354.amzn2.x86_64"
],
"kubernetes.container.image": [
"docker.elastic.co/beats/filebeat:7.13.3"
],
"kubernetes.pod.name": [
"filebeat-beat-filebeat-7s5mw"
],
"host.id": [
"8101459f96f52aa8d2c2bf8a41b09ad4"
],
"kubernetes.pod.ip": [
"n.n.n.n"
],
"kubernetes.namespace_labels.kustomize_toolkit_fluxcd_io/namespace": [
"flux-system"
],
"kubernetes.container.name": [
"filebeat"
],
"host.os.codename": [
"Core"
],
"kubernetes.node.labels.node_kubernetes_io/lifecycle": [
"spot"
],
"kubernetes.labels.pod-template-generation": [
"2"
],
"message": [
"2022-06-21T15:10:43.815Z WARN [elasticsearch] elasticsearch/client.go:408 Cannot index event publisher.Event {
Content:beat.Event{Timestamp:time.Time{wall:0x139a1966, ext:63791421042, loc:(*time.Location)(nil)}, Meta:null,
Fields:{
"agent ":{
"ephemeral_id ": "d8b30f9e-564a-468c-8687-a9aa90719003 ",
"hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"id ": "17af5615-6f68-43b2-9837-e8d39eee7768 ",
"name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"type ": "filebeat ",
"version ": "7.13.3 "},
"cloud ":{
"account ":{
"id ": "redacted "
},
"availability_zone ": "eu-west-1b ",
"image ":{
"id ": "ami-redacted "
},
"instance ":{
"id ": "i-redacted "
},
"machine ":{
"type ": "c5a.xlarge "
},
"provider ": "aws ",
"region ": "eu-west-1 ",
"service ":{
"name ": "EC2 "
}
},
"container ":{
"id ": "6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13 ",
"image ":{
"name ": "redacted.dkr.ecr.eu-west-1.amazonaws.com/redacted:0.0.1943 "
},
"runtime ": "docker "
},
"ecs ":{
"version ": "1.6.0 "},
"error ": " ",
"host ":{
"architecture ": "x86_64 ",
"containerized ":true,
"hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"id ": "8101459f96f52aa8d2c2bf8a41b09ad4 ",
"ip ":[ "n.n.n.n ", "fe80::490:e6ff:fe51:49b9 ", "100.64.16.38 ", "fe80::4df:e1ff:fe25:7ea1 ", "fe80::e04f:cff:fea7:daed ", "fe80::7c6e:5bff:feaf:5835 ", "fe80::34ef:20ff:fe0a:ec9e "],
"mac ":[ "06:90:e6:51:49:b9 ", "06:df:e1:25:7e:a1 ", "e2:4f:0c:a7:da:ed ", "7e:6e:5b:af:58:35 ", "36:ef:20:0a:ec:9e "],
"name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"os ":{
"codename ": "Core ",
"family ": "redhat ",
"kernel ": "5.4.181-99.354.amzn2.x86_64 ",
"name ": "CentOS Linux ",
"platform ": "centos ",
"type ": "linux ",
"version ": "7 (Core) "
}
},
"input ":{
"type ": "container "
},
"kubernetes ":{
"container ":{
"name ": "generic-service-chart "
},
"labels ":{
"app ": "redacted-refined-topics-prd ",
"chart ": "generic-service-chart-0.3.3 ",
"heritage ": "Helm ",
"pod-template-hash ": "5d4b89c5c7 ",
"release ": "redacted-refined-topics "
},
"namespace ": "prd ",
"namespace_labels ":{
"fluxcd_io/sync-gc-mark ": "sha256.bOEJhpO-um8LkeYFneARkw3cGRj0RyYFFxw2W6YOQlQ ",
"kustomize_toolkit_fluxcd_io/name ": "flux-system ",
"kustomize_toolkit_fluxcd_io/namespace ": "flux-system "
},
"namespace_uid ": "df60337d-277e-4506-92d5-3443c843cb69 ",
"node ":{
"hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"labels ":{
"app ": "hpa ",
"beta_kubernetes_io/arch ": "amd64 ",
"beta_kubernetes_io/instance-type ": "c5a.xlarge ",
"beta_kubernetes_io/os ": "linux ",
"failure-domain_beta_kubernetes_io/region ": "eu-west-1 ",
"failure-domain_beta_kubernetes_io/zone ": "eu-west-1b ",
"k8s_amazonaws_com/eniConfig ": "eu-west-1b-eni ",
"kubernetes_io/arch ": "amd64 ",
"kubernetes_io/hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"kubernetes_io/os ": "linux ",
"node_kubernetes_io/instance-type ": "c5a.xlarge ",
"node_kubernetes_io/lifecycle ": "spot ",
"topology_kubernetes_io/region ": "eu-west-1 ",
"topology_kubernetes_io/zone ": "eu-west-1b ",
"vpc_amazonaws_com/eniConfig ": "eu-west-1b-eni "
},
"name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
"uid ": "6f063c79-a3af-4a83-ba49-36dd0cdf90c5 "},
"pod ":{
"ip ": "100.64.26.102 ",
"name ": "redacted-refined-topics-generic-service-chartml4vq ",
"uid ": "db74d802-072a-4ea7-be1f-dcae44bace26 "
},
"replicaset ":{
"name ": "redacted-refined-topics-generic-service-chart-5d4b89c5c7 "
}
},
"level ": "warning ",
"log ":{
"file ":{
"path ": "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log "
},
"offset ":61846
},
"log.level ": "warning ",
"logger ": "src.serializers.mixins ",
"message ": "Expected error for UDA value description. Setting to default value. ",
"sku ":138349918,
"stream ": "stdout ",
"timestamp ": "2022-06-21T15:10:42.328613Z "
},
Private:file.State{
Id: "native::197191726-66305 ",
PrevId: " ",
Finished:false,
Fileinfo:(*os.fileStat)(0xc0017daa90),
Source: "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log ",
Offset:62265,
Timestamp:time.Time{wall:0xc0a495bc870e96e8,
ext:67579503195443,
loc:(*time.Location)(0x55b2e8a9cdc0)},
TTL:-1,
Type: "container ", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xbc0e82e, Device:0x10301},
IdentifierName: "native "}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {
"type ": "mapper_parsing_exception ", "reason ": "object mapping for [error] tried to parse field [error] as object, but found a concrete value "}"
],
"kubernetes.node.labels.kubernetes_io/hostname": [
"ip-n-n-n-n.eu-west-1.compute.internal"
],
"cloud.image.id": [
"ami-redacted"
],
"kubernetes.node.labels.beta_kubernetes_io/arch": [
"amd64"
],
"@timestamp": [
"2022-06-21T15:10:43.816Z"
],
"cloud.account.id": [
"redacted"
],
"host.os.platform": [
"centos"
],
"kubernetes.labels.controller-revision-hash": [
"6ffdbb7864"
],
"log.file.path": [
"/var/log/containers/filebeat-beat-filebeat-7s5mw_devops_filebeat-7c2cea4c00f9836b405118e8f8166d0320f9abae3004c77c67192626e0375e39.log"
],
"kubernetes.labels.beat_k8s_elastic_co/name": [
"filebeat"
],
"agent.ephemeral_id": [
"d8b30f9e-564a-468c-8687-a9aa90719003"
],
"kubernetes.node.labels.kubernetes_io/arch": [
"amd64"
],
"kubernetes.labels.common_k8s_elastic_co/type": [
"beat"
]
},
"highlight": {
"message": [
"2022-06-21T15:10:43.815Z WARN [elasticsearch] elasticsearch/client.go:408 @kibana-highlighted-field@Cannot@/kibana-highlighted-field@ @kibana-highlighted-field@index@/kibana-highlighted-field@ @kibana-highlighted-field@event@/kibana-highlighted-field@ @kibana-highlighted-field@publisher.Event@/kibana-highlighted-field@{Content:beat.Event{Timestamp:time.Time{wall:0x139a1966, ext:63791421042, loc:(*time.Location)(nil)}, Meta:null, Fields:{ "agent ":{ "ephemeral_id ": "d8b30f9e-564a-468c-8687-a9aa90719003 ", "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "id ": "17af5615-6f68-43b2-9837-e8d39eee7768 ", "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "type ": "filebeat ", "version ": "7.13.3 "}, "cloud ":{ "account ":{ "id ": "redacted "}, "availability_zone ": "eu-west-1b ", "image ":{ "id ": "ami-redacted "}, "instance ":{ "id ": "i-redacted "}, "machine ":{ "type ": "c5a.xlarge "}, "provider ": "aws ", "region ": "eu-west-1 ", "service ":{ "name ": "EC2 "}}, "container ":{ "id ": "6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13 ", "image ":{ "name ": "redacted.dkr.ecr.eu-west-1.amazonaws.com/redacted:0.0.1943 "}, "runtime ": "docker "}, "ecs ":{ "version ": "1.6.0 "}, "error ": " ", "host ":{ "architecture ": "x86_64 ", "containerized ":true, "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "id ": "8101459f96f52aa8d2c2bf8a41b09ad4 ", "ip ":[ "n.n.n.n ", "fe80::490:e6ff:fe51:49b9 ", "100.64.16.38 ", "fe80::4df:e1ff:fe25:7ea1 ", "fe80::e04f:cff:fea7:daed ", "fe80::7c6e:5bff:feaf:5835 ", "fe80::34ef:20ff:fe0a:ec9e "], "mac ":[ "06:90:e6:51:49:b9 ", "06:df:e1:25:7e:a1 ", "e2:4f:0c:a7:da:ed ", "7e:6e:5b:af:58:35 ", "36:ef:20:0a:ec:9e "], "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "os ":{ "codename ": "Core ", "family ": "redhat ", "kernel ": "5.4.181-99.354.amzn2.x86_64 ", "name ": "CentOS Linux ", "platform ": "centos ", "type ": "linux ", "version ": "7 (Core) "}}, "input ":{ "type ": "container "}, "kubernetes ":{ "container ":{ "name ": "generic-service-chart "}, "labels ":{ "app ": "redacted-refined-topics-prd ", "chart ": "generic-service-chart-0.3.3 ", "heritage ": "Helm ", "pod-template-hash ": "5d4b89c5c7 ", "release ": "redacted-refined-topics "}, "namespace ": "prd ", "namespace_labels ":{ "fluxcd_io/sync-gc-mark ": "sha256.bOEJhpO-um8LkeYFneARkw3cGRj0RyYFFxw2W6YOQlQ ", "kustomize_toolkit_fluxcd_io/name ": "flux-system ", "kustomize_toolkit_fluxcd_io/namespace ": "flux-system "}, "namespace_uid ": "df60337d-277e-4506-92d5-3443c843cb69 ", "node ":{ "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "labels ":{ "app ": "hpa ", "beta_kubernetes_io/arch ": "amd64 ", "beta_kubernetes_io/instance-type ": "c5a.xlarge ", "beta_kubernetes_io/os ": "linux ", "failure-domain_beta_kubernetes_io/region ": "eu-west-1 ", "failure-domain_beta_kubernetes_io/zone ": "eu-west-1b ", "k8s_amazonaws_com/eniConfig ": "eu-west-1b-eni ", "kubernetes_io/arch ": "amd64 ", "kubernetes_io/hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "kubernetes_io/os ": "linux ", "node_kubernetes_io/instance-type ": "c5a.xlarge ", "node_kubernetes_io/lifecycle ": "spot ", "topology_kubernetes_io/region ": "eu-west-1 ", "topology_kubernetes_io/zone ": "eu-west-1b ", "vpc_amazonaws_com/eniConfig ": "eu-west-1b-eni "}, "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "uid ": "6f063c79-a3af-4a83-ba49-36dd0cdf90c5 "}, "pod ":{ "ip ": "100.64.26.102 ", "name ": "redacted-refined-topics-generic-service-chartml4vq ", "uid ": "db74d802-072a-4ea7-be1f-dcae44bace26 "}, "replicaset ":{ "name ": "redacted-refined-topics-generic-service-chart-5d4b89c5c7 "}}, "level ": "warning ", "log ":{ "file ":{ "path ": "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log "}, "offset ":61846}, "log.level ": "warning ", "logger ": "src.serializers.mixins ", "message ": "Expected error for UDA value description. Setting to default value. ", "sku ":138349918, "stream ": "stdout ", "timestamp ": "2022-06-21T15:10:42.328613Z "}, Private:file.State{Id: "native::197191726-66305 ", PrevId: " ", Finished:false, Fileinfo:(*os.fileStat)(0xc0017daa90), Source: "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log ", Offset:62265, Timestamp:time.Time{wall:0xc0a495bc870e96e8, ext:67579503195443, loc:(*time.Location)(0x55b2e8a9cdc0)}, TTL:-1, Type: "container ", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xbc0e82e, Device:0x10301}, IdentifierName: "native "}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): { "type ": "mapper_parsing_exception ", "reason ": "object mapping for [error] tried to parse field [error] as object, but found a concrete value "}"
]
},
"sort": [
1655824243816
]
}
There is clearly some mapping or manipulation of the JSON configured somewhere, but I do not know where. Also I wondered if the mapper_parsing_exception explains why this is happening.
There have been no recent updates or changes to the configuration.
I am stumped as to where to look or how to understand why this is occurring.
Any help/pointers is/are greatly appreciated.
The version of stack is:
Filebeat 7.17.1
ES 7.17.1
Kibana 7.17.1
Cheers, JP
