Metricbeat -> redis -> logstash -> elasticsearch: failed to parse field [kubernetes.labels.app]

[asp]

Hi,

not sure if this issue is better be posted here or in metricbeat section.

I have deployed metricbeat in kubernetes using this manual:
https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html
Now I am trying to get kubernetes metrics to elasticsearch

I changed following things:
-> Output is changed redis, not elasticsearch.

Logstash is pulling from redis and directly pushing to elasticsearch, without any filters. I am just using redis as message broker for all other logs, etc. So I want to keep same route for all data which is coming to elasticsearch.

In Logstash I can see following errors:

[2019-08-13T08:46:19,364][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-7.3.0-2019.w33", :_type=>"_doc", :routing=>nil}, #<LogStash::Event:0x4b443107>], :response=>{"index"=>{"_index"=>"metricbeat-7.3.0-2019.w33", "_type"=>"_doc", "_id"=>"PLIqimwBthAGD_OhXAJU", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'PLIqimwBthAGD_OhXAJU'. Preview of field's value: '{kubernetes={io/part-of=ingress-nginx, io/name=ingress-nginx}}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:450"}}}}}

So why can logstash not parse the field. Output of metricbeat to redis looks like plain json, so what problem does logstash have?

Is there any other best practice to ship metricbeat data via redis to elasticsearch?

Thanks, Andreas

[Badger]

That's really an elasticsearch question. I do not understand what triggered that mapper exception, but someone in the elasticsearch forum might.

[asp]

moved to elasticsearch

[asp]

One idea about this topic.

If I check the labels directly in kubernetes, I have these:

Labels:             app.kubernetes.io/name=ingress-nginx
                    app.kubernetes.io/part-of=ingress-nginx
                    pod-template-hash=86449c74bb

I have other pods where the label is set to

app=logstash

I don't know, if the error message is matching for this possible root cause in my mind.
The failing label has dots in the name. May it be that metricbeat is trying to save this as structure ?
meaning:

app:
  - kubernetes: 
    - io/name: "ingress-nginx"

That would collide to previously label app: "logstash" -> trying to save object in string / keyword.
But I don't know, if that error message fits.

Any Ideas to solve this issue except for changing my own labels to avoid this conflict or to use this (in my opinion too long) structure app.kubernetes.io/name=logstash?

[asp]

Can you confirm that metricbeat is creating a structure of dotted label names?
Is there any flag that can be used in configuration, that metricbeat is storing the label name as string?
Is there anything like labels.dedot: false for kubernetes module? at least it is not listed in reference.yaml

[asp]

So here the results of my research:

yes, if labels have dots in their name, they will be interpreted as structure. This may fail on indexing when you have different labels like app=myapp and app.kubernetes.io/name=ingress-nginx. They are conflicting in elasticsearch because the first one is a string, the second one is a structure.

I found this here: https://github.com/elastic/beats/pull/9939

This is quite the solution. setting dedot options for labels and anotations are fixing the issue. Then the . will be replaced with _ in field names.