Mapper [kubernetes.labels.app] of different type, current_type [text], merged_type [ObjectMapper]


(Steve Szabo) #1

We get this error in elastic:

java.lang.IllegalArgumentException: mapper [kubernetes.labels.app] of different type, current_type [text], merged_type [ObjectMapper]

The message seems to have kubernetes.labels.app set to a string:

sszabo@tor976568e1 [/home/sszabo/tmp] $ cat log | jq -r .kubernetes
{
  "namespace": "spinnaker",
  "container": {
    "name": "clouddriver"
  },
  "pod": {
    "name": "spin-clouddriver-59648c5449-snt2x"
  },
  "labels": {
    "app": "spin",
    "cluster": "spin-clouddriver",
    "app.kubernetes.io/managed-by": "halyard",
    "app.kubernetes.io/name": "clouddriver",
    "pod-template-hash": "1520471005",
    "app.kubernetes.io/part-of": "spinnaker",
    "app.kubernetes.io/version": "1.9.5"
  }
}

(Alexander Reelsen) #2

Elasticsearch uses dots in fieldnames internally to create an object tree. In the above case labels.app and labels.app.kubernetes clash, as one is a string and the other creates an inner object. Try app.name instead of app and this should work.


(Steve Szabo) #3

These labels are set by spinnaker, I have no control over them:

sszabo@tor976568e1 [/home/sszabo] $ kubectl get -o yaml pods spin-clouddriver-69768cbb5b-pqthp | yq -y .metadata.labels
app: spin
app.kubernetes.io/managed-by: halyard
app.kubernetes.io/name: clouddriver
app.kubernetes.io/part-of: spinnaker
app.kubernetes.io/version: 1.9.5
cluster: spin-clouddriver
pod-template-hash: '2532476616'

(Alexander Reelsen) #4

you could change the JSON structure before indexing or is that not possible? You havent mentioned a lot about the ingesting infrastructure.


(Steve Szabo) #5

These events are generated by spinnaker pods running under kubernetes. Spinnaker is setting the pod labels which are sent to logstash via filebeat.


(Alexander Reelsen) #6

Then you need to have a mechanism to change this data before indexing.

Either via an Elasticsearch pipeline in an Ingest Node or via (https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html)[beats] - you have not mentioned you method of ingestion, that might be helpful in this case.

--Alex


(Steve Szabo) #7

I'll look at using logstash to remove the app label, thanks!


(Steve Szabo) #8

I was able to rename the label from logstash:

filter {

    if [kubernetes] {
            
        if [kubernetes][namespace] == "spinnaker" {
            mutate {
            rename => {"[kubernetes][labels][app]" => "[kubernetes][labels][spinapp]"}
            }
        }

    }

(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.