Filebeat : Split the multiline json(array of json object) from message field to separate fields

[Magesh_02 (Magesh)]

I need to use filebeat to push my json data into Elasticsearch, but I'm having trouble decoding my json fields into separate fields extracted from the message field.

Filebeat version : 7.16.2

Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: "2"
      target: ""
      overwrite_keys: true
      add_error_key: false

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true
  hosts: ["http://localhost:9200"] 

Json Input :

{
  "Source": [
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 150325
    },
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 106026
    }
  ]
}

Current Output:

Expected Output:
separate fields

_source: {
@timestamp: "2022-01-12T09:12:36.904Z",
"date": "28-09-2021",
 "language": " C++",
 "comment": 11,
  "code": 106026
input: {
type: "log"
},

Please suggest to decode the multiline json in filebeat?

[ChrsMark (Chris Mark)]

How about using json options right after the multiline ones?

[Magesh_02 (Magesh)]

Hi @ChrsMark

i tried your suggestion, still no effect

filebeat.inputs:

• type: log

  enabled: true

  paths:

  • /logs/*.json

  multiline.pattern: '^{'

  multiline.negate: true

  multiline.match: after

json.keys_under_root: true

json.add_error_key: true

json.message_key: "log"

processors:

• decode_json_fields:

  fields: ["message"]

  process_array: false

  max_depth: "2"

  target: ""

  overwrite_keys: true

  add_error_key: false

output.elasticsearch:

Boolean flag to enable or disable the output module.

enabled: true

hosts: ["http://localhost:9200"]