Decode_json_fields and array

[s17n (srm)]

Hello,

It seems decode_json_fields can't decode array.
This is my log :

{"level":"panic","application":"command","stack":[{"func":"main.func1","line":"51","source":"main.go"},{"func":"gopanic","line":"965","source":"panic.go"},{"func":"panicmem","line":"212","source":"panic.go"},{"func":"sigpanic","line":"734","source":"signal_unix.go"},{"func":"main","line":"92","source":"main.go"},{"func":"main","line":"225","source":"proc.go"},{"func":"goexit","line":"1371","source":"asm_amd64.s"}],"error":"runtime error: invalid memory address or nil pointer dereference","caller":"/app/command/main.go:51","time":"2021-08-16T09:49:58Z","message":"Unexpected error"}

(This log is one line)

My config :

      - decode_json_fields:
          fields: ["message"]
          target: ""
          overwrite_keys: true
          add_error_key: true
          process_array: true

All fields have been decoded properly, except the field "stack" :

stack.func: main.func1, gopanic, panicmem, sigpanic, main, main, goexit
stack.line: 51, 965, 212, 734, 92, 225, 1371
stack.source: main.go, panic.go, panic.go, signal_unix.go, main.go, proc.go, asm_amd64.s

stack should be decoded like that :

"stack":[
{
"func":"main.func1",
"line":"51",
"source":"main.go"
},
{
"func":"gopanic",
"line":"965",
"source":"panic.go"
},
{
"func":"panicmem",
"line":"212",
"source":"panic.go"
},
{
"func":"sigpanic",
"line":"734",
"source":"signal_unix.go"
},
{
"func":"main",
"line":"92",
"source":"main.go"
},
{
"func":"main",
"line":"225",
"source":"proc.go"
},
{
"func":"goexit",
"line":"1371",
"source":"asm_amd64.s"
}
],

[andrewkroh]

Filebeat will output the data similar to how you posted that it should.

I assume you are looking at the data in Elasticsearch. If you look at the _source field for the document you will see what Filebeat sent.

The reason arrays are flattened like that in the data returned by Elasticsearch is due to the data type used by default. You can change how the data is stored by changing the data type used for this field. See Nested field type | Elasticsearch Guide [7.14] | Elastic for details.

One way to add your own mapping for this field is to use Filebeats setup.template.append_fields option (docs). You have to delete your current index and run filebeat setup for the change to take effect.

[s17n (srm)]

Good catch thank you