Decode_json_fields.fields: ["message"] decodes JSON logs with "logs"

[olivierwa (Olivier C)]

Hello,
I am running an Elastic+Kibana+Filebeat docker stack to monitor containers. Everything is working fine but there is something that I do not understand and wasted many hours on it to debug.

The log files under /var/lib/docker/containers are in this format:

{"log":"{\"asctime\": \"2019-09-02 15:07:45,420\", \"filename\": \"_internal.py\", \"funcName\": \"_log\", \"levelname\": \"INFO\", \"lineno\": 122, \"message\": \" * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)\", \"name\": \"werkzeug\", \"thread\": 140275406490472, \"threadName\": \"MainThread\"}\r\n","stream":"stdout","time":"2019-09-02T15:07:45.4207396Z"}

Essentially it is a JSON with 3 top fields:

{
"log": {...},
"stream:": "stdout",
"time":
}

So at the root, the JSON does not contain any "message" field. Yet in order to decode this, my filebeat configuration needs to indicate:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - config:
            - type: container
              format: docker
              paths:
                - "/var/lib/docker/containers/${data.docker.container.id}/*.log"
              processors:
                - decode_json_fields:
                    **fields: ["message"]**
                    target: ""
                    overwrite_keys: true         

What does this "message" mean? What is the relation with fields:["message"] and the log file ? since the log file does not contain any message field.

Thanks,
Olivier