sszabo
(Steve Szabo)
October 16, 2018, 6:04pm
1
We get this error in elastic:
java.lang.IllegalArgumentException: mapper [kubernetes.labels.app] of different type, current_type [text], merged_type [ObjectMapper]
The message seems to have kubernetes.labels.app set to a string:
sszabo@tor976568e1 [/home/sszabo/tmp] $ cat log | jq -r .kubernetes
{
"namespace": "spinnaker",
"container": {
"name": "clouddriver"
},
"pod": {
"name": "spin-clouddriver-59648c5449-snt2x"
},
"labels": {
"app": "spin",
"cluster": "spin-clouddriver",
"app.kubernetes.io/managed-by": "halyard",
"app.kubernetes.io/name": "clouddriver",
"pod-template-hash": "1520471005",
"app.kubernetes.io/part-of": "spinnaker",
"app.kubernetes.io/version": "1.9.5"
}
}
spinscale
(Alexander Reelsen)
October 17, 2018, 7:53am
2
Elasticsearch uses dots in fieldnames internally to create an object tree. In the above case labels.app
and labels.app.kubernetes
clash, as one is a string and the other creates an inner object. Try app.name
instead of app
and this should work.
sszabo
(Steve Szabo)
October 17, 2018, 3:56pm
3
These labels are set by spinnaker, I have no control over them:
sszabo@tor976568e1 [/home/sszabo] $ kubectl get -o yaml pods spin-clouddriver-69768cbb5b-pqthp | yq -y .metadata.labels
app: spin
app.kubernetes.io/managed-by: halyard
app.kubernetes.io/name: clouddriver
app.kubernetes.io/part-of: spinnaker
app.kubernetes.io/version: 1.9.5
cluster: spin-clouddriver
pod-template-hash: '2532476616'
spinscale
(Alexander Reelsen)
October 17, 2018, 8:46pm
4
you could change the JSON structure before indexing or is that not possible? You havent mentioned a lot about the ingesting infrastructure.
sszabo
(Steve Szabo)
October 19, 2018, 4:50pm
5
These events are generated by spinnaker pods running under kubernetes. Spinnaker is setting the pod labels which are sent to logstash via filebeat.
spinscale
(Alexander Reelsen)
October 22, 2018, 8:32am
6
Then you need to have a mechanism to change this data before indexing.
Either via an Elasticsearch pipeline in an Ingest Node or via (https://www.elastic.co/guide/en/beats/filebeat/current/rename-fields.html )[beats] - you have not mentioned you method of ingestion, that might be helpful in this case.
--Alex
sszabo
(Steve Szabo)
October 23, 2018, 12:49pm
7
I'll look at using logstash to remove the app label, thanks!
sszabo
(Steve Szabo)
October 23, 2018, 3:34pm
8
I was able to rename the label from logstash:
filter {
if [kubernetes] {
if [kubernetes][namespace] == "spinnaker" {
mutate {
rename => {"[kubernetes][labels][app]" => "[kubernetes][labels][spinapp]"}
}
}
}
system
(system)
Closed
November 20, 2018, 3:37pm
9
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.