Using kv and Grok Pattern in Ingest Pipelines

[geek876]

I am trying to extract a log-pattern

[LEVEL] (COMPONENT) MESSAGE [msgid: MESSAGE ID]

example message:

[INFO   ] (core.workflow) Start scan, workflow_id='lms::workflow::WorkflowExecutor(0x259c99f94b0, name = "WF-127.0.0.1:60708")', dataId='4d66c5069365476a8f36431e6369ab0b', fileName='winlogbeat.yml' [msgid: 3491]

In the pipeline I have two filters:

(1) Grok: That extracts level, component, kvmessage and mid - This works.
(2) kv: I want to now split values like workflow_id, name etc as key value pairs. However, this is not working. Would appreciate some assistance.

    [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "\\[%{WORD:level}\\s*\\]: \\(%{COMPONENT:component}\\) %{MSG:kvmessage}\\[%{GREEDYDATA:mid}\\]"
          ],
          "pattern_definitions": {
            "COMPONENT": "\\w*.\\w*",
            "STATUS": "\\w*\\s\\w*",
            "MSG": "[^\\[]*"
          },
          "ignore_missing": true
        }
      },
      {
        "kv": {
          "field": "kvmessage",
          "field_split": ",",
          "value_split": "="
        }
      }
    ]