I have an application running in docker that logs in json format to stdout. Then I have filebeat (also running in docker) shipping container logs. Now I'm wondering i it's possible to to decode the application logs and append the properties to the root of the object containing the docker metadata. The docs say neither yay or nay but I haven't been able to make it work. I have however been able to decode logs in other formats by adding a suitable label co.elastic.logs/processors.dissect.tokenizer. This is why I tried adding co.elastic.logs/processors.decode_json_fields.fields. No success. I've also had success with modules (i.e. co.elastic.logs/module: "elasticsearch") to ship decoded logs with this setup.
I think it would make sense to introduce a new hint for this, in the same way we have some to do multiline, I could see these working good for your case:
Thanks for the reply. I did some testing around with the processor hint but never got it working. The expected format for the fields hint was a bit unclear. If I didn't put it in a pair of brackets and quotes["message"] I'd get errors.
What I ended up doing was to come up with a label of my own json_logger-label and then in the filbeats config:
This worked very fine but then I had some other issues with auto-discover so I ended up disabling it.
I liked the hints-based approach though. I could put config in the docker-compose file and have the config be there together with other variables.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.