Extracting some JSON fields from the message

[hs_shaikh (Hamza Shaikh)]

Hello there, I want to extract "applicationOwner" and "proxyResponseCode" JSON fields from my message

My message filed look like this:

message": "TID: [-1234] [2022-02-07 18:59:15,667] INFO {org.wso2.am.analytics.publisher.sample.reporter.LogCounterMetric} - Metric Name: apim:response Metric Value: {proxyResponseCode=500, errorType=null, applicationOwner=admin, , apiType=HTTP}"

I tried JSON source but it's not working

Error parsing json {:source=>"message" <LogStash::Json::ParserError: Unrecognized token 'TID': was expecting ('true', 'false' or 'null')

[sholzhauer]

Hi, you will have to use two parsers/processors.
Below example is in logstash:

filter {
  grok {
    match => [ "TID: \[%{DATA}\] \[%{DATA\] \[%{DATA:timestamp}\] %{WORD:log_lvl} %{GREEDYDATA:json_src}" ]
  }

  json {
    source => "json_src"
  }
}

This will first parse the line and take the non json part out, or more accuratly put the json part in its own field. You can then use the json parser on that field.

This example puts all fields at the root btw.

[hs_shaikh (Hamza Shaikh)]

Hi @sholzhauer I tried the above solution but it's giving me grok plugin error, I tried updating it as :-

grok {
    match =>{ "message" => [ "TID: [%{DATA}] [%{DATA}] [%{DATA:timestamp}] %{WORD:log_lvl} %{GREEDYDATA:json_src}" ]
  }
}
  json {
    source => "json_src"
  }

But it doesn't change anything

[sholzhauer]

I put one \[%{DATA}\] to much in the pattern, it should be

TID: \[%{DATA}\] \[%{DATA:timestamp}\] %{WORD:log_lvl} %{GREEDYDATA:json_src}

[hs_shaikh (Hamza Shaikh)]

Hi @sholzhauer thanks for the pattern it created a field as

json_src": {org.wso2.am.analytics.publisher.sample.reporter.LogCounterMetric} - Metric Name: apim:response Metric Value: {proxyResponseCode=500, errorType=null, applicationOwner=admin, , apiType=HTTP}

How can I extract applicationOwner from that and create a new field?

[sholzhauer]

You do so by using the second part:

json {
  source => "json_src"
}

This will extract the json to the root level. Although i'm not entirely sure if this will correctly parse correctly due to the whitespaces and = instead of :

[Badger]

You could try

    grok { match => { "message" => "{(?<[@metadata][kvData]>[^}]+)}$" } }
    kv { source => "[@metadata][kvData]" field_split => "," trim_key => " " }

which will produce

          "apiType" => "HTTP",
"proxyResponseCode" => "500",
        "errorType" => "null",
 "applicationOwner" => "admin"