Oddness with decode_json_fields and json.keys_under_root

[LollerAgent]

Given the following configuration:

filebeat.prospectors:
- type: docker
  containers.ids:
    - ea97d09738e4bdf7b41cb926ab1276110ed3ce93f73f005d4f5da540ef0928c1
  processors:
  - add_docker_metadata: ~
  - decode_json_fields:
      fields: ["message"]
      target: "log"

The message field is decoded into the log (target) field, but the original message field remains as an encoded string at the root of the JSON object:

  "log": {
    "message": "",
    "level": "info",
    "distributed_trace_id": "",
    "http": {
      "request": {
        "remote_ip": "173.197.145.174",
        "method": "GET",
        "path": "/stylesheets/fonts/WhitneySSm-BookItalic.ttf",
        "host": ""
      },
      "response": {
        "status_code": 404
      }
    },
    "legacy_message": "ts=2018-05-21T20:48:51.880970898Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=173.197.145.174"
  },
...
  "message": "{\"legacy_message\":\"ts=2018-05-21T20:48:51.880970898Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=173.197.145.174\",\"message\":\"\",\"level\":\"info\",\"distributed_trace_id\":\"\",\"http\":{\"request\":{\"method\":\"GET\",\"path\":\"/stylesheets/fonts/WhitneySSm-BookItalic.ttf\",\"host\":\"\",\"remote_ip\":\"173.197.145.174\"},\"response\":{\"status_code\":404}}}",

If I add json.keys_under_root: false (which is false by default, right?) - the message field gets decoded underneath a new json key (not log as defined by the target) but the original message key gets deleted:

  "json": {
    "legacy_message": "ts=2018-05-21T20:46:41.807431938Z method=GET path=/stylesheets/fonts/WhitneySSm-BookItalic.ttf host= remote_ip=173.197.145.174",
    "message": "",
    "level": "info",
    "distributed_trace_id": "",
    "http": {
      "request": {
        "method": "GET",
        "path": "/stylesheets/fonts/WhitneySSm-BookItalic.ttf",
        "host": "",
        "remote_ip": "173.197.145.174"
      },
      "response": {
        "status_code": 404
      }
    }
  },

What is the correct behavior here? This seems odd. My goal is to get the decoded message field under a new log key, but delete the original message field that includes the encoded JSON. Is this possible? This is in 6.2.4.

[jsoriano (Jaime Soriano)]

Indeed this behaviour is a bit odd, but this may be caused by some tricky behaviour of the different json parsing options, that are intended for different use cases.

For what you are trying to do, you can keep using the decode_json_fields option, and to delete the original message field add the drop_fields processor, like this:

 - drop_fields:
     fields: ["message"]