# Ingest mixed container logs with text and JSON \[filebeat\]\[docker\]

**URL:** <https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139>\
**Category:** Logs\
**Created:** [April 24, 2021, 12:22pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139 "2021-04-24T12:22:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://discuss.elastic.co/u/bluepuma77)\
**Post date:** [April 24, 2021, 12:22pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/1 "2021-04-24T12:22:05Z")

</div>

Hi all,

we are currently using plain text logging and import container output with filebeat into elastic and kibana.

We would like to migrate to structured logging with JSON. Is a bit-by-bit migration possible, can I have text log and JSON in the container output and still see both in elastic/kibana?

Regards  
bluepuma

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [April 28, 2021, 9:35am UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/2 "2021-04-28T09:35:37Z")

</div>

Yep, that's possible using two different templates for the same container and route logs based on whether or not they start with `{`.

Here's an example of how it should work with k8s:

```auto
filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            contains:
              kubernetes.container.name: "my-app-1"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
              include_lines: ['^{']
              json.keys_under_root: true
              json.overwrite_keys: true
              json.add_error_key: true
              json.expand_keys: true
        - condition:
            contains:
              kubernetes.container.name: "my-app-1"
          config:
            - type: container
              paths:
                - "/var/log/containers/*-${data.kubernetes.container.id}.log"
              multiline.pattern: '^[[:blank:]]'
              multiline.negate: false
              multiline.match: after
              exclude_lines: ['^{']

```

Disclaimer: I have not tested the config. Please let me know if it works for you or if you had to make adjustments.

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://discuss.elastic.co/u/bluepuma77)\
**Post date:** [April 28, 2021, 8:10pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/3 "2021-04-28T20:10:35Z")

</div>

We don't use kubernetes and have about 50 docker containers, so 2 configurations per container is not really feasible.

@felixbarny: **Feature request** : new config option `json.add_error_content: key`

If a line can not be JSON-parsed (error), then it is just added as text under `key`.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [April 29, 2021, 6:48am UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/4 "2021-04-29T06:48:39Z")

</div>

You can either set up your conditions so that they match on a particular label or add multiple conditions to the same template for all of your apps: [Multiple conditions with autodiscover & docker containers - #2 by steffens](https://discuss.elastic.co/t/multiple-conditions-with-autodiscover-docker-containers/153634/2)

There's also a docker-based autodiscover so the k8s-style autodiscover example above would look like this for docker:

```yaml
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.labels: "log-format-json-and-text"
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              include_lines: ['^{']
              json.keys_under_root: true
              json.overwrite_keys: true
              json.add_error_key: true
              json.expand_keys: true
        - condition:
            contains:
              docker.container.labels: "log-format-json-and-text"
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log
              exclude_lines: ['^{']
              multiline.pattern: '^[[:blank:]]'
              multiline.negate: false
              multiline.match: after

```

See [Autodiscover | Filebeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_docker_2) for more info about autodiscover.

> [@bluepuma77](#):
>
> @felixbarny: **Feature request** : new config option `json.add_error_content: key`
> 
> If a line can not be JSON-parsed (error), then it is just added as text under `key` .

What's not working with `json.add_error_key: true`? Is the unparseable JSON under the wrong key or is it completely absent from the indexed document?

---

<div class="post-metadata">

**Author:** ![bluepuma77](https://avatars.discourse-cdn.com/v4/letter/b/a9adbd/32.png) [@bluepuma77](https://discuss.elastic.co/u/bluepuma77)\
**Post date:** [May 12, 2021, 1:22pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/5 "2021-05-12T13:22:05Z")

</div>

Interesting, after removing `json.message_key` it just works for me 😃

```auto
docker run \
  --rm \
  --name testXYZ \
  --label co.elastic.logs/enabled=true \
  --label co.elastic.logs/json.keys_under_root=true \
  --label co.elastic.logs/json.add_error_key=false \
  python:3-slim python -c 'import time; print("{\"message\":\"testX\", \"session\":\"testY\"}\ntestZ", flush=True); time.sleep(10);'

```

From log:

```auto
    {"message":"testX", "session":"testY"}
    testZ

```

To elastic:

```auto
    { "message": "testX", "session": "testY", ... }
    { "message": "testZ", ... }

```

For a JSON line the `message` is set, additional attributes, too. And for text just the `message`.  
👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 9, 2021, 1:22pm UTC](https://discuss.elastic.co/t/ingest-mixed-container-logs-with-text-and-json-filebeat-docker/271139/6 "2021-06-09T13:22:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.