Parse JSON in "message" field

[surprised_ferret]

Elastic version 7.11.1

• How do I parse a JSON structure (nested, one field has an array) out into separate fields of their own?

I'm running filebeat on my k8 instance, this is what my "filebeat.yml" value looks like, inside the filebeat-kubernetes.yaml file.

I think I am missing something but the documentation isn't very clear.

I have looked at Filebeat parse json and Filebeat JSON message but they are using something called filebeat.prospectors which looks a bit like filebeat.inputs but different.

I have used the decode_json_fields key because I looked at this doc: Decode JSON fields | Filebeat Reference [7.11] | Elastic

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - decode_json_fields:
            fields: ["message"]
            max_depth: 8

the message JSON i want to parse out looks like this

{"@timestamp":"2021-03-19T19:46:00.2675696+00:00","level":"Information","messageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","message":"Executing ObjectResult, writing value of type '\"Microsoft.AspNetCore.Mvc.ProblemDetails\"'.","fields":{"Type":"Microsoft.AspNetCore.Mvc.ProblemDetails","EventId":{"Id":1,"Name":"ObjectResultExecuting"},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"caefff34-0ffd-4aa0-81a6-b68c86df21e5","ActionName":"etc_api.AcknowledgeTvlController.AcknowledgeTvl (etc-api)","RequestId":"0HM758EQLA55U:00000002","RequestPath":"/rest/v1/tvl/acknowledgement","SpanId":"4902e7163ebd8441","TraceId":"0c5a930c74bc7247b2c006002de2602b","ParentId":"0000000000000000","ConnectionId":"0HM758EQLA55U"}}

[Andre_Letterer (André Letterer)]

Hi,

Prospectors were famous in version 6.x and are now sunset a longer time ago towards inputs:

Brewing in Beats: Rename Filebeat prospectors to inputs | Elastic Blog

Additionally the decode_json_fields processor is not the right place as this would mean one field has containing json. But here the whole message is in JSON embedded and for that this particular documentation is the right place:

Container input | Filebeat Reference [7.11] | Elastic

So you could configure your snippet like that:

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      json.add_error_key: true
      json.keys_under_root: true
      json.overwrite_keys: true
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

And then add whatever json.xxxx settings you additionally might need.