Mapper [kubernetes.labels.app] of different type, current_type [text], merged_type [ObjectMapper]

We get this error in elastic:

java.lang.IllegalArgumentException: mapper [kubernetes.labels.app] of different type, current_type [text], merged_type [ObjectMapper]

The message seems to have kubernetes.labels.app set to a string:

sszabo@tor976568e1 [/home/sszabo/tmp] $ cat log | jq -r .kubernetes
{
  "namespace": "spinnaker",
  "container": {
    "name": "clouddriver"
  },
  "pod": {
    "name": "spin-clouddriver-59648c5449-snt2x"
  },
  "labels": {
    "app": "spin",
    "cluster": "spin-clouddriver",
    "app.kubernetes.io/managed-by": "halyard",
    "app.kubernetes.io/name": "clouddriver",
    "pod-template-hash": "1520471005",
    "app.kubernetes.io/part-of": "spinnaker",
    "app.kubernetes.io/version": "1.9.5"
  }
}

Elasticsearch uses dots in fieldnames internally to create an object tree. In the above case labels.app and labels.app.kubernetes clash, as one is a string and the other creates an inner object. Try app.name instead of app and this should work.

These labels are set by spinnaker, I have no control over them:

sszabo@tor976568e1 [/home/sszabo] $ kubectl get -o yaml pods spin-clouddriver-69768cbb5b-pqthp | yq -y .metadata.labels
app: spin
app.kubernetes.io/managed-by: halyard
app.kubernetes.io/name: clouddriver
app.kubernetes.io/part-of: spinnaker
app.kubernetes.io/version: 1.9.5
cluster: spin-clouddriver
pod-template-hash: '2532476616'

you could change the JSON structure before indexing or is that not possible? You havent mentioned a lot about the ingesting infrastructure.

These events are generated by spinnaker pods running under kubernetes. Spinnaker is setting the pod labels which are sent to logstash via filebeat.

Then you need to have a mechanism to change this data before indexing.

Either via an Elasticsearch pipeline in an Ingest Node or via (https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html)[beats] - you have not mentioned you method of ingestion, that might be helpful in this case.

--Alex

I'll look at using logstash to remove the app label, thanks!

I was able to rename the label from logstash:

filter {

    if [kubernetes] {
            
        if [kubernetes][namespace] == "spinnaker" {
            mutate {
            rename => {"[kubernetes][labels][app]" => "[kubernetes][labels][spinapp]"}
            }
        }

    }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.