Why does the decode_json_fields processor not have a keys_under_root option?

Elastic Stack Beats
1

I want to parse logs in the following format. But what confuses me is that i have to specify fields for the decode_json_fields processor under which to find json strings. But what if there is no outer field for the json? How would the following log to be parsed?

{
"deploymentid": "3ae9ef41-ce1b-435a-abc9-4b441a4914bd",
"level": "ERROR",
"time": "2019-03-21T22:14:31.475878214Z",
"api": {
"name": "PutObject",
"args": {
"bucket": "test",
"object": "issue"
}
},
"remotehost": "127.0.0.1",
"requestID": "158E190E7B258FDB",
"userAgent": "Minio (linux; amd64) minio-go/v6.0.20 mc/2019-03-15T20:48:52Z",
"error": {
"message": "read tcp 172.16.1.105:9000->172.16.1.103:44678: read: connection reset by peer",
"source": [
"cmd/object-handlers.go:1073:cmd.objectAPIHandlers.PutObjectHandler()",
"cmd/api-router.go:79:cmd.objectAPIHandlers.PutObjectHandler-fm()",
"net/http/server.go:1964:http.HandlerFunc.ServeHTTP()"
],
"variables": {
"peerAddress": "127.0.0.1:9000"
}
}
}

2

What decode json does is that it takes a field from the input and then parse its content.
so if you have

{
  "msg": {your_content}
}

and configuration

{
    "fields":        ["msg"],
    "process_array": false,
    "max_depth":     2,
    "target":        "myfield",
}

it will take a msg field, decode the content and then create a myfield with parsed out json object.
so something like this

{
   "msg": {your content},
   "myfield": {
      "deploymenyId": "3ae...",
      "level": "ERROR"
      "api": {
          ...
      }
   }
}

if your log entry is json itself maybe this may come in handy

3

Does that mean in my case, i have to combine the json and the multiline options in the prospector, like for example:

json.keys_under_root: true
json.overwrite_keys: true

multiline.pattern: '^{'
multiline.negate: true
multiline.match: after

processors:

  • decode_json_fields:
    fields: ["api", "args", "error"]
    process_array: true
    max_depth: 3
    target: ""
    overwrite_keys: true
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.