# Ingest mixed container logs with text and JSON \[ECK\]\[filebeat\]

**URL:** <https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 17, 2021, 2:45pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464 "2021-11-17T14:45:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sven\_Eliasson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sven_eliasson/32/77470_2.png) [@Sven\_Eliasson](https://discuss.elastic.co/u/Sven_Eliasson)\
**Post date:** [November 17, 2021, 2:45pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464/1 "2021-11-17T14:45:30Z")

</div>

Hi,

I'm trying to digest logs with Filebeat on our Kubernetes cluster which is using ECK on Kubernetes. We are migrating logs to a JSON format, but many legacy or 3rd party containers use plain logs. So we need to support both in the long run.

I came across another topic covering this exact challenge, but couldn't make it work with the suggested solution:

> [@Ingest mixed container logs with text and JSON \[filebeat\]\[docker\]](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139):
>
> Hi all, we are currently using plain text logging and import container output with filebeat into elastic and kibana. We would like to migrate to structured logging with JSON. Is a bit-by-bit migration possible, can I have text log and JSON in the container output and still see both in elastic/kibana? Regards bluepuma

I'm pretty new in Filebeat and the whole Elasticstack - so I may have trouble understanding the nuances in the configuration. The topic suggests using "exclude\_lines" to split JSON and non-JSON logs.

Filebeat config:

```auto
  config:
    filebeat.inputs:
      - include_lines:
          - '^{'
        json.add_error_key: 'true'
        json.expand_keys: 'true'
        json.keys_under_root: 'true'
        json.overwrite_keys: 'true'
        paths:
          - /var/log/containers/*.log
        type: container
      - exclude_lines:
          - '^{'
        paths:
          - /var/log/containers/*.log
        type: container

```

This config throws an error, that I need to add a "message\_key" when using exclude.

Based on my understanding I set it to "log" since the docker container logs are a JSON themselves and "log" contains the JSON string.

The result is, that I don't see any of my JSON longs in Elasticstack, but rather a bunch of JSON parse errors.

```auto
ERROR	[reader_json]	readjson/json.go:74	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

```

In case I set json.add\_error\_key: 'false' - I can see the non-JSON logs, but not my JSON logs.

So in short:

- exclude\_lines / include\_lines doesnt seem to work like it should.
- I dont realy understand why the "message\_key" is needed to exclude the line - isnt type:container already unwrapping the "log" field? In that case - why do I need it at all?

---

<div class="post-metadata">

**Author:** ![matschaffer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matschaffer/32/95396_2.png) [@matschaffer](https://discuss.elastic.co/u/matschaffer)\
**Post date:** [November 18, 2021, 2:48am UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464/2 "2021-11-18T02:48:50Z")

</div>

Welcome to the forums, Sven!

I've moved this to the beats topic where I think it'll get a better answer.

I haven't tried this myself but I wonder if setting the input to text, then using [Decode JSON fields | Filebeat Reference [7.15] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html) would be the way to go.

You could probably use a `when` (if you can determine which containers send json) or setting `add_error_key: false` such that non-json logs just get sent out without processing.

---

<div class="post-metadata">

**Author:** ![matschaffer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matschaffer/32/95396_2.png) [@matschaffer](https://discuss.elastic.co/u/matschaffer)\
**Post date:** [November 18, 2021, 2:54am UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464/3 "2021-11-18T02:54:06Z")

</div>

I found [Ingest mixed container logs with text and JSON [filebeat][docker] - #2 by felixbarny](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/2) which has a config that might work in your case that's closer to what you were trying with include/exclude but at the autodiscover-level.

@felixbarny generally knows his stuff so maybe try that before my processor idea 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 16, 2021, 2:54am UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-eck-filebeat/289464/4 "2021-12-16T02:54:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.