Ingest mixed container logs with text and JSON [ECK][filebeat]

[Sven_Eliasson]

Hi,

I'm trying to digest logs with Filebeat on our Kubernetes cluster which is using ECK on Kubernetes. We are migrating logs to a JSON format, but many legacy or 3rd party containers use plain logs. So we need to support both in the long run.

I came across another topic covering this exact challenge, but couldn't make it work with the suggested solution:

Ingest mixed container logs with text and JSON [filebeat][docker]

Hi all, we are currently using plain text logging and import container output with filebeat into elastic and kibana. We would like to migrate to structured logging with JSON. Is a bit-by-bit migration possible, can I have text log and JSON in the container output and still see both in elastic/kibana? Regards bluepuma

I'm pretty new in Filebeat and the whole Elasticstack - so I may have trouble understanding the nuances in the configuration. The topic suggests using "exclude_lines" to split JSON and non-JSON logs.

Filebeat config:

  config:
    filebeat.inputs:
      - include_lines:
          - '^{'
        json.add_error_key: 'true'
        json.expand_keys: 'true'
        json.keys_under_root: 'true'
        json.overwrite_keys: 'true'
        paths:
          - /var/log/containers/*.log
        type: container
      - exclude_lines:
          - '^{'
        paths:
          - /var/log/containers/*.log
        type: container

This config throws an error, that I need to add a "message_key" when using exclude.

Based on my understanding I set it to "log" since the docker container logs are a JSON themselves and "log" contains the JSON string.

The result is, that I don't see any of my JSON longs in Elasticstack, but rather a bunch of JSON parse errors.

ERROR	[reader_json]	readjson/json.go:74	Error decoding JSON: json: cannot unmarshal number into Go value of type map[string]interface {}

In case I set json.add_error_key: 'false' - I can see the non-JSON logs, but not my JSON logs.

So in short:

• exclude_lines / include_lines doesnt seem to work like it should.
• I dont realy understand why the "message_key" is needed to exclude the line - isnt type:container already unwrapping the "log" field? In that case - why do I need it at all?

[matschaffer]

Welcome to the forums, Sven!

I've moved this to the beats topic where I think it'll get a better answer.

I haven't tried this myself but I wonder if setting the input to text, then using Decode JSON fields | Filebeat Reference [7.15] | Elastic would be the way to go.

You could probably use a when (if you can determine which containers send json) or setting add_error_key: false such that non-json logs just get sent out without processing.

[matschaffer]

I found Ingest mixed container logs with text and JSON [filebeat][docker] - #2 by felixbarny which has a config that might work in your case that's closer to what you were trying with include/exclude but at the autodiscover-level.

@felixbarny generally knows his stuff so maybe try that before my processor idea