How to parse json from message field

[wrender1 (Wes)]

Hello, I'm looking for help with parsing json data out of a log field. I'm using the elastic agent standalone in Kubernetes and not sure how to configure it for this. I believe the filebeat portion of the elastic agent is indexing the messages. Some of the fields I would like to index as a numeric integer if possible. Right now the events come in as something like this from the stdout of the containers and are pretty much useless in Kibana to build visualizations around:

  "message": [
    "I0104 15:47:24.388872   118 SyslogReporter.cpp:10] APP-METRIC {\"fields\":{\"drop\":0},\"measurement\":\"core.datafield\",\"tags\":{\"host\":\"blahblah-testing-9rtvp\",\"instance\":\"0\",\"source\":\"core.datafield\",\"type\":\"data\",\"worker\":\"0\"},\"time\":\"2024-01-04T15:47:24.000000000Z\"}"

Ideally I want to match based on "APP-METRIC" and strip out the json below the "fields" key.

Thanks!

[jessgarson]

Hi, welcome to the community @wrender1.

There are a few similar forum posts on this subject that might be helpful to look at here:

• Extracting the JSON fields from the message - #2 by Raed
• Parse json in Log field to get individual fields for visualization - #7 by aaron-nimocks
• Extracting some JSON fields from the message - #7 by Badger

Best,
Jessica

[wrender1 (Wes)]

Thanks for the links. Since I was using Elastic Agent, I was able to do it with this processor:

processors:
  - if:
      regexp:
        message: "^.*APP-METRIC.*$"     
    then:
    # Rename fields entry as it seems to conflict with elastic field names
    - replace:
        fields:
          - field: "message"
            pattern: "fields"
            replacement: "myappname"
        ignore_missing: true
        fail_on_error: false
    # Strip out anything in the message field left of APP-METRIC as it is not wanted
    - replace:
        fields:
          - field: "message"
            pattern: "^.*APP-METRIC"
            replacement: ""
        ignore_missing: true
        fail_on_error: false
    # Decode the remaining message field to json
    - decode_json_fields:
        fields: ["message"]
        process_array: true
        max_depth: 1
        target: "myappstats"
        overwrite_keys: false
        add_error_key: false

[jessgarson]

Thanks for sharing your solution, @wrender1.