# Metricbeat -\> redis -\> logstash -\> elasticsearch: failed to parse field \[kubernetes.labels.app\] **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [August 13, 2019, 9:21am UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992 "2019-08-13T09:21:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [August 13, 2019, 9:21am UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/1 "2019-08-13T09:21:09Z")
Hi, not sure if this issue is better be posted here or in metricbeat section. I have deployed metricbeat in kubernetes using this manual: [https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html](https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html) Now I am trying to get kubernetes metrics to elasticsearch I changed following things: -\> Output is changed redis, not elasticsearch. Logstash is pulling from redis and directly pushing to elasticsearch, without any filters. I am just using redis as message broker for all other logs, etc. So I want to keep same route for all data which is coming to elasticsearch. In Logstash I can see following errors: `[2019-08-13T08:46:19,364][WARN][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-7.3.0-2019.w33", :_type=>"_doc", :routing=>nil}, #], :response=>{"index"=>{"_index"=>"metricbeat-7.3.0-2019.w33", "_type"=>"_doc", "_id"=>"PLIqimwBthAGD_OhXAJU", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [kubernetes.labels.app] of type [keyword] in document with id 'PLIqimwBthAGD_OhXAJU'. Preview of field's value: '{kubernetes={io/part-of=ingress-nginx, io/name=ingress-nginx}}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:450"}}}}}` So why can logstash not parse the field. Output of metricbeat to redis looks like plain json, so what problem does logstash have? Is there any other best practice to ship metricbeat data via redis to elasticsearch? Thanks, Andreas ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 13, 2019, 1:06pm UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/2 "2019-08-13T13:06:24Z")
That's really an elasticsearch question. I do not understand what triggered that mapper exception, but someone in the elasticsearch forum might. ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [August 13, 2019, 2:06pm UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/3 "2019-08-13T14:06:56Z")
moved to elasticsearch ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [August 14, 2019, 6:49am UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/4 "2019-08-14T06:49:44Z")
One idea about this topic. If I check the labels directly in kubernetes, I have these: ``` Labels: app.kubernetes.io/name=ingress-nginx app.kubernetes.io/part-of=ingress-nginx pod-template-hash=86449c74bb ``` I have other pods where the label is set to ``` app=logstash ``` I don't know, if the error message is matching for this possible root cause in my mind. The failing label has dots in the name. May it be that metricbeat is trying to save this as structure ? meaning: ``` app: - kubernetes: - io/name: "ingress-nginx" ``` That would collide to previously label app: "logstash" -\> trying to save object in string / keyword. But I don't know, if that error message fits. Any Ideas to solve this issue except for changing my own labels to avoid this conflict or to use this (in my opinion too long) structure [app.kubernetes.io/name=logstash](http://app.kubernetes.io/name=logstash)? ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [August 14, 2019, 10:14am UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/5 "2019-08-14T10:14:10Z")
Can you confirm that metricbeat is creating a structure of dotted label names? Is there any flag that can be used in configuration, that metricbeat is storing the label name as string? Is there anything like labels.dedot: false for kubernetes module? at least it is not listed in reference.yaml ---
**Author:** ![asp](https://avatars.discourse-cdn.com/v4/letter/a/9fc348/32.png) [@asp](https://discuss.elastic.co/u/asp)\ **Post date:** [September 9, 2019, 12:01pm UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/6 "2019-09-09T12:01:00Z")
So here the results of my research: yes, if labels have dots in their name, they will be interpreted as structure. This may fail on indexing when you have different labels like app=myapp and [app.kubernetes.io/name=ingress-nginx](http://app.kubernetes.io/name=ingress-nginx). They are conflicting in elasticsearch because the first one is a string, the second one is a structure. I found this here: [https://github.com/elastic/beats/pull/9939](https://github.com/elastic/beats/pull/9939) This is quite the solution. setting dedot options for labels and anotations are fixing the issue. Then the `.` will be replaced with `_` in field names. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 7, 2019, 12:01pm UTC](https://discuss.elastic.co/t/metricbeat-redis-logstash-elasticsearch-failed-to-parse-field-kubernetes-labels-app/194992/7 "2019-10-07T12:01:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.