# Kubernetes: Filebeat parses JSON in message field no matter if I want or not

**URL:** <https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 21, 2023, 3:30pm UTC](https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089 "2023-02-21T15:30:18Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomx1](https://avatars.discourse-cdn.com/v4/letter/t/779978/32.png) [@tomx1](https://discuss.elastic.co/u/tomx1)\
**Post date:** [February 21, 2023, 3:30pm UTC](https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089/1 "2023-02-21T15:30:19Z")

</div>

I want to use Filebeat (current version) to collect logs from our Kubernetes Cluster by using this manual: [Run Filebeat on Kubernetes | Filebeat Reference [8.6] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html)

I want to control if the message of a cointainer should be parsed as json or not due two reasons:

1.) Not every Pod in our Cluster is logging in Json  
2.) I want to make sure that only ECS compliant Json is parsed

My favorite solution is to configure filebeat to output to a remote logstash elastic\_agent input (one reason for that is that I can manipulate messages in logstash easily if necessary). Sending the events to a logstash instance is working without a problem BUT if I use this configuration below in filebeat to gather the logfiles on the kubernetes node, the message field is automatically parsed (NOT using annotations):

```auto
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

```

If I remove this part:  
`${data.kubernetes.container.id}`  
So that the input path is just:  
`- /var/log/containers/*.log`

Its not being parsed anymore, but filebeat seems to mix up messages one container exposes with informations of other containers running in the cluster. So overall its not useable, and I wonder what am I doing wrong?

In short: To me it looks like adding `${data.kubernetes.container.id}` in the path automatically activates the json parser.

Additionaly, its strange that an error.message and error.type field is added with the following values when the message field is automatically parsed:

 ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91e6f51b3856a8511a60c89c3fd4a74d120f8925.png)

---

<div class="post-metadata">

**Author:** ![tomx1](https://avatars.discourse-cdn.com/v4/letter/t/779978/32.png) [@tomx1](https://discuss.elastic.co/u/tomx1)\
**Post date:** [February 22, 2023, 9:43am UTC](https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089/2 "2023-02-22T09:43:47Z")

</div>

I really can't imagine that nobody stumbled over this yet. It seems just not possbile with filebeat in a kubernetes cluster to:

1.) NOT parse json a pod potentially exposes if using autodiscover of type kubernetes AND using `${data.kubernetes.container.id}` in the container input path (as it is suggested by documentation)

2.) removing `${data.kubernetes.container.id}` in the filepath fucks up the whole logging, as filebeat mixes log messages/metadata of different pods togheter randomly

By the way, setting the json settings does not make a difference either way. Filebeat is still adding the fields at root level if message contains a json:

```auto
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
            json.keys_under_root: false
            json.add_error_key: false
            json.message_key: message

```

---

<div class="post-metadata">

**Author:** ![tomx1](https://avatars.discourse-cdn.com/v4/letter/t/779978/32.png) [@tomx1](https://discuss.elastic.co/u/tomx1)\
**Post date:** [February 23, 2023, 8:45am UTC](https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089/3 "2023-02-23T08:45:15Z")

</div>

sorry for the noise, I've found the issue it was a misconfiguration

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 23, 2023, 10:45am UTC](https://discuss.elastic.co/t/kubernetes-filebeat-parses-json-in-message-field-no-matter-if-i-want-or-not/326089/4 "2023-03-23T10:45:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.