Decode json logs filebeat docker hints-based autodiscover


Can I have filebeat via hint-based autodiscover ship my json-formatted logs like:

        "@timestamp": "2019-06-15T19:48:04.963Z",
        "foo": "bar"

instead of:

        "@timestamp": "2019-06-15T19:48:04.963Z",
        "message": "{\"foo\":\"bar\"}"

I have an application running in docker that logs in json format to stdout. Then I have filebeat (also running in docker) shipping container logs. Now I'm wondering i it's possible to to decode the application logs and append the properties to the root of the object containing the docker metadata. The docs say neither yay or nay but I haven't been able to make it work. I have however been able to decode logs in other formats by adding a suitable label co.elastic.logs/processors.dissect.tokenizer. This is why I tried adding co.elastic.logs/processors.decode_json_fields.fields. No success. I've also had success with modules (i.e. co.elastic.logs/module: "elasticsearch") to ship decoded logs with this setup.

I have a git repo reproducing my setup here

Hi @goekboet,

I think it would make sense to introduce a new hint for this, in the same way we have some to do multiline, I could see these working good for your case:


I have created a new issue to track this, feel free to send your comments or subscribe for updates!

Also as a different way of doing this, you could use the processors hint to leverage decode_json_fields, for example, something like this should work:

docker run -l co.elastic.logs/processors.0.decode_json_fields.fields=message -l co.elastic.logs/"" -it busybox echo '{"foo":"bar"}'

Thanks for the reply. I did some testing around with the processor hint but never got it working. The expected format for the fields hint was a bit unclear. If I didn't put it in a pair of brackets and quotes["message"] I'd get errors.
What I ended up doing was to come up with a label of my own json_logger-label and then in the filbeats config:

        - condition:
              docker.container.labels.json_logger: "True"
            - type: docker
                - "${}"
              json.keys_under_root: true
              json.add_error_key: true

This worked very fine but then I had some other issues with auto-discover so I ended up disabling it.
I liked the hints-based approach though. I could put config in the docker-compose file and have the config be there together with other variables.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.