Decode json logs filebeat docker hints-based autodiscover

[goekboet (Erik Göök)]

TL;DR;

Can I have filebeat via hint-based autodiscover ship my json-formatted logs like:

 {
        "@timestamp": "2019-06-15T19:48:04.963Z",
        "foo": "bar"
    } 

instead of:

    {
        "@timestamp": "2019-06-15T19:48:04.963Z",
        "message": "{\"foo\":\"bar\"}"
    }

I have an application running in docker that logs in json format to stdout. Then I have filebeat (also running in docker) shipping container logs. Now I'm wondering i it's possible to to decode the application logs and append the properties to the root of the object containing the docker metadata. The docs say neither yay or nay but I haven't been able to make it work. I have however been able to decode logs in other formats by adding a suitable label co.elastic.logs/processors.dissect.tokenizer. This is why I tried adding co.elastic.logs/processors.decode_json_fields.fields. No success. I've also had success with modules (i.e. co.elastic.logs/module: "elasticsearch") to ship decoded logs with this setup.

I have a git repo reproducing my setup here

[exekias (Carlos Pérez Aradros)]

Hi @goekboet,

I think it would make sense to introduce a new hint for this, in the same way we have some to do multiline, I could see these working good for your case:

co.elastic.logs/json.keys_under_root
co.elastic.logs/json.add_error_key
co.elastic.logs/json.message_key

I have created a new issue to track this, feel free to send your comments or subscribe for updates! https://github.com/elastic/beats/issues/12634

[exekias (Carlos Pérez Aradros)]

Also as a different way of doing this, you could use the processors hint to leverage decode_json_fields, for example, something like this should work:

docker run -l co.elastic.logs/processors.0.decode_json_fields.fields=message -l co.elastic.logs/processors.0.decode_json_fields.target="" -it busybox echo '{"foo":"bar"}'

[goekboet (Erik Göök)]

Thanks for the reply. I did some testing around with the processor hint but never got it working. The expected format for the fields hint was a bit unclear. If I didn't put it in a pair of brackets and quotes["message"] I'd get errors.
What I ended up doing was to come up with a label of my own json_logger-label and then in the filbeats config:

templates:
        - condition:
            equals:
              docker.container.labels.json_logger: "True"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              json.keys_under_root: true
              json.add_error_key: true

This worked very fine but then I had some other issues with auto-discover so I ended up disabling it.
I liked the hints-based approach though. I could put config in the docker-compose file and have the config be there together with other variables.