Mapper parsing exception - Am I looking in the right place or is ES fooling me!?

Hi all

Apologies if I don't have all the correct terminology as I am new to the elastic stack.

I have just been allocated a task to remedy this problem;

We have a Filebeat - ES - Kibana installation

When users run the following:

https://kibana/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(),filters:!(),index:bc055660-c693-11ea-bcb9-5b73a837b97b,interval:auto,query:(language:kuery,query:'%22Cannot%20index%20event%20publisher.Event%22'),sort:!(!('@timestamp',desc)))

Apparently we used to get results back that included a documents/indexes like:

Expected error for UDA value description. Setting to default value.

now we get back this (which includes the message above):

{
  "_index": "filebeat-7.13.3-2022.06.19-000407",
  "_type": "_doc",
  "_id": "V1XQhoEB-VCaPvu79D9z",
  "_version": 1,
  "_score": null,
  "fields": {
    "kubernetes.labels.beat_k8s_elastic_co/version": [
      "7.13.3"
    ],
    "kubernetes.node.uid": [
      "6f063c79-a3af-4a83-ba49-36dd0cdf90c5"
    ],
    "kubernetes.node.labels.app": [
      "hpa"
    ],
    "kubernetes.namespace_uid": [
      "4bf344df-36ff-489d-83de-95bf03348ac9"
    ],
    "kubernetes.labels.beat_k8s_elastic_co/config-checksum": [
      "2fca66b447485134295d2ed965f1e5e3cd244f384e25bccab18c21b6"
    ],
    "host.os.name.text": [
      "CentOS Linux"
    ],
    "kubernetes.node.labels.topology_kubernetes_io/zone": [
      "eu-west-1b"
    ],
    "host.hostname": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "host.mac": [
      "06:90:e6:51:49:b9",
      "06:df:e1:25:7e:a1",
      "e2:4f:0c:a7:da:ed",
      "7e:6e:5b:af:58:35",
      "36:ef:20:0a:ec:9e"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "container.id": [
      "7c2cea4c00f9836b405118e8f8166d0320f9abae3004c77c67192626e0375e39"
    ],
    "cloud.availability_zone": [
      "eu-west-1b"
    ],
    "kubernetes.node.labels.k8s_amazonaws_com/eniConfig": [
      "eu-west-1b-eni"
    ],
    "container.image.name": [
      "docker.elastic.co/beats/filebeat:7.13.3"
    ],
    "host.os.version": [
      "7 (Core)"
    ],
    "kubernetes.node.labels.vpc_amazonaws_com/eniConfig": [
      "eu-west-1b-eni"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace_labels.kustomize_toolkit_fluxcd_io/name": [
      "flux-system"
    ],
    "kubernetes.namespace": [
      "devops"
    ],
    "host.os.name": [
      "CentOS Linux"
    ],
    "agent.name": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "host.name": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "kubernetes.node.labels.topology_kubernetes_io/region": [
      "eu-west-1"
    ],
    "host.os.type": [
      "linux"
    ],
    "cloud.region": [
      "eu-west-1"
    ],
    "kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone": [
      "eu-west-1b"
    ],
    "input.type": [
      "container"
    ],
    "log.offset": [
      6027077
    ],
    "agent.hostname": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "kubernetes.namespace_labels.fluxcd_io/sync-gc-mark": [
      "sha256.VJgxYJtEwMtOG1t8L1iOzJ1Ci-nKIEYkx21huD372xM"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "cloud.machine.type": [
      "c5a.xlarge"
    ],
    "cloud.provider": [
      "aws"
    ],
    "container.runtime": [
      "docker"
    ],
    "agent.id": [
      "17af5615-6f68-43b2-9837-e8d39eee7768"
    ],
    "cloud.service.name": [
      "EC2"
    ],
    "ecs.version": [
      "1.8.0"
    ],
    "host.containerized": [
      true
    ],
    "kubernetes.node.labels.beta_kubernetes_io/instance-type": [
      "c5a.xlarge"
    ],
    "agent.version": [
      "7.13.3"
    ],
    "host.os.family": [
      "redhat"
    ],
    "kubernetes.node.name": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "kubernetes.node.labels.failure-domain_beta_kubernetes_io/region": [
      "eu-west-1"
    ],
    "kubernetes.node.hostname": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "kubernetes.node.labels.node_kubernetes_io/instance-type": [
      "c5a.xlarge"
    ],
    "kubernetes.pod.uid": [
      "61dc703a-d35c-493a-ba50-aac93aa18b48"
    ],
    "host.ip": [
      "n.n.n.n",
      "fe80::490:e6ff:fe51:49b9",
      "100.64.16.38",
      "fe80::4df:e1ff:fe25:7ea1",
      "fe80::e04f:cff:fea7:daed",
      "fe80::7c6e:5bff:feaf:5835",
      "fe80::34ef:20ff:fe0a:ec9e"
    ],
    "cloud.instance.id": [
      "i-redacted"
    ],
    "agent.type": [
      "filebeat"
    ],
    "stream": [
      "stderr"
    ],
    "host.os.kernel": [
      "5.4.181-99.354.amzn2.x86_64"
    ],
    "kubernetes.container.image": [
      "docker.elastic.co/beats/filebeat:7.13.3"
    ],
    "kubernetes.pod.name": [
      "filebeat-beat-filebeat-7s5mw"
    ],
    "host.id": [
      "8101459f96f52aa8d2c2bf8a41b09ad4"
    ],
    "kubernetes.pod.ip": [
      "n.n.n.n"
    ],
    "kubernetes.namespace_labels.kustomize_toolkit_fluxcd_io/namespace": [
      "flux-system"
    ],
    "kubernetes.container.name": [
      "filebeat"
    ],
    "host.os.codename": [
      "Core"
    ],
    "kubernetes.node.labels.node_kubernetes_io/lifecycle": [
      "spot"
    ],
    "kubernetes.labels.pod-template-generation": [
      "2"
    ],
    "message": [
      "2022-06-21T15:10:43.815Z  WARN  [elasticsearch]  elasticsearch/client.go:408  Cannot index event publisher.Event {
          Content:beat.Event{Timestamp:time.Time{wall:0x139a1966, ext:63791421042, loc:(*time.Location)(nil)}, Meta:null, 
          Fields:{
             "agent ":{
             "ephemeral_id ": "d8b30f9e-564a-468c-8687-a9aa90719003 ",
             "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
             "id ": "17af5615-6f68-43b2-9837-e8d39eee7768 ",
             "name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
             "type ": "filebeat ",
             "version ": "7.13.3 "},
             "cloud ":{
               "account ":{
                 "id ": "redacted "
              },
               "availability_zone ": "eu-west-1b ",
               "image ":{
                 "id ": "ami-redacted "
              },
               "instance ":{
                 "id ": "i-redacted "
              },
               "machine ":{
                 "type ": "c5a.xlarge "
              },
               "provider ": "aws ",
               "region ": "eu-west-1 ",
               "service ":{
                 "name ": "EC2 "
              }
            },
             "container ":{
               "id ": "6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13 ",
               "image ":{
                 "name ": "redacted.dkr.ecr.eu-west-1.amazonaws.com/redacted:0.0.1943 "
              },
               "runtime ": "docker "
            },
             "ecs ":{
               "version ": "1.6.0 "},
               "error ": " ",
               "host ":{
                 "architecture ": "x86_64 ",
                 "containerized ":true,
                 "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
                 "id ": "8101459f96f52aa8d2c2bf8a41b09ad4 ",
                 "ip ":[ "n.n.n.n ", "fe80::490:e6ff:fe51:49b9 ", "100.64.16.38 ", "fe80::4df:e1ff:fe25:7ea1 ", "fe80::e04f:cff:fea7:daed ", "fe80::7c6e:5bff:feaf:5835 ", "fe80::34ef:20ff:fe0a:ec9e "],
                 "mac ":[ "06:90:e6:51:49:b9 ", "06:df:e1:25:7e:a1 ", "e2:4f:0c:a7:da:ed ", "7e:6e:5b:af:58:35 ", "36:ef:20:0a:ec:9e "],
                 "name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
                 "os ":{
                   "codename ": "Core ",
                   "family ": "redhat ",
                   "kernel ": "5.4.181-99.354.amzn2.x86_64 ",
                   "name ": "CentOS Linux ",
                   "platform ": "centos ",
                   "type ": "linux ",
                   "version ": "7 (Core) "
                }
              },
               "input ":{
                 "type ": "container "
              },
               "kubernetes ":{
                 "container ":{
                   "name ": "generic-service-chart "
                },
                 "labels ":{
                   "app ": "redacted-refined-topics-prd ",
                   "chart ": "generic-service-chart-0.3.3 ",
                   "heritage ": "Helm ",
                   "pod-template-hash ": "5d4b89c5c7 ",
                   "release ": "redacted-refined-topics "
                },
                 "namespace ": "prd ",
                 "namespace_labels ":{
                   "fluxcd_io/sync-gc-mark ": "sha256.bOEJhpO-um8LkeYFneARkw3cGRj0RyYFFxw2W6YOQlQ ",
                   "kustomize_toolkit_fluxcd_io/name ": "flux-system ",
                   "kustomize_toolkit_fluxcd_io/namespace ": "flux-system "
                },
                 "namespace_uid ": "df60337d-277e-4506-92d5-3443c843cb69 ",
                 "node ":{
                   "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
                   "labels ":{
                     "app ": "hpa ",
                     "beta_kubernetes_io/arch ": "amd64 ",
                     "beta_kubernetes_io/instance-type ": "c5a.xlarge ",
                     "beta_kubernetes_io/os ": "linux ",
                     "failure-domain_beta_kubernetes_io/region ": "eu-west-1 ",
                     "failure-domain_beta_kubernetes_io/zone ": "eu-west-1b ",
                     "k8s_amazonaws_com/eniConfig ": "eu-west-1b-eni ",
                     "kubernetes_io/arch ": "amd64 ",
                     "kubernetes_io/hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ",
                     "kubernetes_io/os ": "linux ",
                     "node_kubernetes_io/instance-type ": "c5a.xlarge ",
                     "node_kubernetes_io/lifecycle ": "spot ",
                     "topology_kubernetes_io/region ": "eu-west-1 ",
                     "topology_kubernetes_io/zone ": "eu-west-1b ",
                     "vpc_amazonaws_com/eniConfig ": "eu-west-1b-eni "
                  },
                   "name ": "ip-n-n-n-n.eu-west-1.compute.internal ",
                   "uid ": "6f063c79-a3af-4a83-ba49-36dd0cdf90c5 "},
                   "pod ":{
                     "ip ": "100.64.26.102 ",
                     "name ": "redacted-refined-topics-generic-service-chartml4vq ",
                     "uid ": "db74d802-072a-4ea7-be1f-dcae44bace26 "
                  },
                   "replicaset ":{
                     "name ": "redacted-refined-topics-generic-service-chart-5d4b89c5c7 "
                  }
                },
                 "level ": "warning ",
                 "log ":{
                   "file ":{
                     "path ": "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log "
                  },
                   "offset ":61846
                },
                 "log.level ": "warning ",
                 "logger ": "src.serializers.mixins ",
                 "message ": "Expected error for UDA value description. Setting to default value. ",
                 "sku ":138349918,
                 "stream ": "stdout ",
                 "timestamp ": "2022-06-21T15:10:42.328613Z "
              }, 
              Private:file.State{
                Id: "native::197191726-66305 ", 
                PrevId: " ", 
                Finished:false, 
                Fileinfo:(*os.fileStat)(0xc0017daa90), 
                Source: "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log ", 
                Offset:62265, 
                Timestamp:time.Time{wall:0xc0a495bc870e96e8, 
                  ext:67579503195443, 
                  loc:(*time.Location)(0x55b2e8a9cdc0)}, 
                  TTL:-1, 
                  Type: "container ", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xbc0e82e, Device:0x10301}, 
                  IdentifierName: "native "}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {
                     "type ": "mapper_parsing_exception ", "reason ": "object mapping for [error] tried to parse field [error] as object, but found a concrete value "}"
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "ip-n-n-n-n.eu-west-1.compute.internal"
    ],
    "cloud.image.id": [
      "ami-redacted"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "@timestamp": [
      "2022-06-21T15:10:43.816Z"
    ],
    "cloud.account.id": [
      "redacted"
    ],
    "host.os.platform": [
      "centos"
    ],
    "kubernetes.labels.controller-revision-hash": [
      "6ffdbb7864"
    ],
    "log.file.path": [
      "/var/log/containers/filebeat-beat-filebeat-7s5mw_devops_filebeat-7c2cea4c00f9836b405118e8f8166d0320f9abae3004c77c67192626e0375e39.log"
    ],
    "kubernetes.labels.beat_k8s_elastic_co/name": [
      "filebeat"
    ],
    "agent.ephemeral_id": [
      "d8b30f9e-564a-468c-8687-a9aa90719003"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "kubernetes.labels.common_k8s_elastic_co/type": [
      "beat"
    ]
  },
  "highlight": {
    "message": [
      "2022-06-21T15:10:43.815Z  WARN  [elasticsearch]  elasticsearch/client.go:408  @kibana-highlighted-field@Cannot@/kibana-highlighted-field@ @kibana-highlighted-field@index@/kibana-highlighted-field@ @kibana-highlighted-field@event@/kibana-highlighted-field@ @kibana-highlighted-field@publisher.Event@/kibana-highlighted-field@{Content:beat.Event{Timestamp:time.Time{wall:0x139a1966, ext:63791421042, loc:(*time.Location)(nil)}, Meta:null, Fields:{ "agent ":{ "ephemeral_id ": "d8b30f9e-564a-468c-8687-a9aa90719003 ", "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "id ": "17af5615-6f68-43b2-9837-e8d39eee7768 ", "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "type ": "filebeat ", "version ": "7.13.3 "}, "cloud ":{ "account ":{ "id ": "redacted "}, "availability_zone ": "eu-west-1b ", "image ":{ "id ": "ami-redacted "}, "instance ":{ "id ": "i-redacted "}, "machine ":{ "type ": "c5a.xlarge "}, "provider ": "aws ", "region ": "eu-west-1 ", "service ":{ "name ": "EC2 "}}, "container ":{ "id ": "6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13 ", "image ":{ "name ": "redacted.dkr.ecr.eu-west-1.amazonaws.com/redacted:0.0.1943 "}, "runtime ": "docker "}, "ecs ":{ "version ": "1.6.0 "}, "error ": " ", "host ":{ "architecture ": "x86_64 ", "containerized ":true, "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "id ": "8101459f96f52aa8d2c2bf8a41b09ad4 ", "ip ":[ "n.n.n.n ", "fe80::490:e6ff:fe51:49b9 ", "100.64.16.38 ", "fe80::4df:e1ff:fe25:7ea1 ", "fe80::e04f:cff:fea7:daed ", "fe80::7c6e:5bff:feaf:5835 ", "fe80::34ef:20ff:fe0a:ec9e "], "mac ":[ "06:90:e6:51:49:b9 ", "06:df:e1:25:7e:a1 ", "e2:4f:0c:a7:da:ed ", "7e:6e:5b:af:58:35 ", "36:ef:20:0a:ec:9e "], "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "os ":{ "codename ": "Core ", "family ": "redhat ", "kernel ": "5.4.181-99.354.amzn2.x86_64 ", "name ": "CentOS Linux ", "platform ": "centos ", "type ": "linux ", "version ": "7 (Core) "}}, "input ":{ "type ": "container "}, "kubernetes ":{ "container ":{ "name ": "generic-service-chart "}, "labels ":{ "app ": "redacted-refined-topics-prd ", "chart ": "generic-service-chart-0.3.3 ", "heritage ": "Helm ", "pod-template-hash ": "5d4b89c5c7 ", "release ": "redacted-refined-topics "}, "namespace ": "prd ", "namespace_labels ":{ "fluxcd_io/sync-gc-mark ": "sha256.bOEJhpO-um8LkeYFneARkw3cGRj0RyYFFxw2W6YOQlQ ", "kustomize_toolkit_fluxcd_io/name ": "flux-system ", "kustomize_toolkit_fluxcd_io/namespace ": "flux-system "}, "namespace_uid ": "df60337d-277e-4506-92d5-3443c843cb69 ", "node ":{ "hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "labels ":{ "app ": "hpa ", "beta_kubernetes_io/arch ": "amd64 ", "beta_kubernetes_io/instance-type ": "c5a.xlarge ", "beta_kubernetes_io/os ": "linux ", "failure-domain_beta_kubernetes_io/region ": "eu-west-1 ", "failure-domain_beta_kubernetes_io/zone ": "eu-west-1b ", "k8s_amazonaws_com/eniConfig ": "eu-west-1b-eni ", "kubernetes_io/arch ": "amd64 ", "kubernetes_io/hostname ": "ip-n-n-n-n.eu-west-1.compute.internal ", "kubernetes_io/os ": "linux ", "node_kubernetes_io/instance-type ": "c5a.xlarge ", "node_kubernetes_io/lifecycle ": "spot ", "topology_kubernetes_io/region ": "eu-west-1 ", "topology_kubernetes_io/zone ": "eu-west-1b ", "vpc_amazonaws_com/eniConfig ": "eu-west-1b-eni "}, "name ": "ip-n-n-n-n.eu-west-1.compute.internal ", "uid ": "6f063c79-a3af-4a83-ba49-36dd0cdf90c5 "}, "pod ":{ "ip ": "100.64.26.102 ", "name ": "redacted-refined-topics-generic-service-chartml4vq ", "uid ": "db74d802-072a-4ea7-be1f-dcae44bace26 "}, "replicaset ":{ "name ": "redacted-refined-topics-generic-service-chart-5d4b89c5c7 "}}, "level ": "warning ", "log ":{ "file ":{ "path ": "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log "}, "offset ":61846}, "log.level ": "warning ", "logger ": "src.serializers.mixins ", "message ": "Expected error for UDA value description. Setting to default value. ", "sku ":138349918, "stream ": "stdout ", "timestamp ": "2022-06-21T15:10:42.328613Z "}, Private:file.State{Id: "native::197191726-66305 ", PrevId: " ", Finished:false, Fileinfo:(*os.fileStat)(0xc0017daa90), Source: "/var/log/containers/redacted-refined-topics-generic-service-chartml4vq_prd_generic-service-chart-6686a4f744c47015006f6eecf12e2ac9b8270a50359f67b18e8ac92dcabaaa13.log ", Offset:62265, Timestamp:time.Time{wall:0xc0a495bc870e96e8, ext:67579503195443, loc:(*time.Location)(0x55b2e8a9cdc0)}, TTL:-1, Type: "container ", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xbc0e82e, Device:0x10301}, IdentifierName: "native "}, TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): { "type ": "mapper_parsing_exception ", "reason ": "object mapping for [error] tried to parse field [error] as object, but found a concrete value "}"
    ]
  },
  "sort": [
    1655824243816
  ]
}

There is clearly some mapping or manipulation of the JSON configured somewhere, but I do not know where. Also I wondered if the mapper_parsing_exception explains why this is happening.

There have been no recent updates or changes to the configuration.

I am stumped as to where to look or how to understand why this is occurring.

Any help/pointers is/are greatly appreciated.

The version of stack is:

Filebeat 7.17.1
ES 7.17.1
Kibana 7.17.1

Cheers, JP

Just in case it is not obvious, we would like the search to return only the single line message rather than the verbose JSON.

Thanks

Welcome to our community! :smiley:

Can you share some screenshots from Kibana?

Thanks @warkolm :sunglasses:

I can get sceenshots, what view(s) do you think would be helpful?

Cheers, JP

Of the Kibana page that shows this error.