Parse JSON in "message" field

Elastic Stack Beats
1

Elastic version 7.11.1

  • How do I parse a JSON structure (nested, one field has an array) out into separate fields of their own?

I'm running filebeat on my k8 instance, this is what my "filebeat.yml" value looks like, inside the filebeat-kubernetes.yaml file.

I think I am missing something but the documentation isn't very clear.

I have looked at Filebeat parse json and Filebeat JSON message but they are using something called filebeat.prospectors which looks a bit like filebeat.inputs but different.

I have used the decode_json_fields key because I looked at this doc: Decode JSON fields | Filebeat Reference [7.11] | Elastic

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - decode_json_fields:
            fields: ["message"]
            max_depth: 8

the message JSON i want to parse out looks like this

{"@timestamp":"2021-03-19T19:46:00.2675696+00:00","level":"Information","messageTemplate":"Executing ObjectResult, writing value of type '{Type}'.","message":"Executing ObjectResult, writing value of type '\"Microsoft.AspNetCore.Mvc.ProblemDetails\"'.","fields":{"Type":"Microsoft.AspNetCore.Mvc.ProblemDetails","EventId":{"Id":1,"Name":"ObjectResultExecuting"},"SourceContext":"Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor","ActionId":"caefff34-0ffd-4aa0-81a6-b68c86df21e5","ActionName":"etc_api.AcknowledgeTvlController.AcknowledgeTvl (etc-api)","RequestId":"0HM758EQLA55U:00000002","RequestPath":"/rest/v1/tvl/acknowledgement","SpanId":"4902e7163ebd8441","TraceId":"0c5a930c74bc7247b2c006002de2602b","ParentId":"0000000000000000","ConnectionId":"0HM758EQLA55U"}}

2

Hi,

Prospectors were famous in version 6.x and are now sunset a longer time ago towards inputs:

Additionally the decode_json_fields processor is not the right place as this would mean one field has containing json. But here the whole message is in JSON embedded and for that this particular documentation is the right place:

So you could configure your snippet like that:

  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      json.add_error_key: true
      json.keys_under_root: true
      json.overwrite_keys: true
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

And then add whatever json.xxxx settings you additionally might need.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.