Filebeat : Split the multiline json(array of json object) from message field to separate fields

Elastic Stack Beats
1

I need to use filebeat to push my json data into Elasticsearch, but I'm having trouble decoding my json fields into separate fields extracted from the message field.

Filebeat version : 7.16.2

Filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
  - /logs/*.json
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match:  after

processors:
  - decode_json_fields:
      fields: ["message"]
      process_array: false
      max_depth: "2"
      target: ""
      overwrite_keys: true
      add_error_key: false

output.elasticsearch:
  # Boolean flag to enable or disable the output module.
  enabled: true
  hosts: ["http://localhost:9200"]

Json Input :

{
  "Source": [
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 150325
    },
    {
      "date": "28-09-2021",
      "language": " C++",
      "comment": 11,
      "code": 106026
    }
  ]
}

Current Output:

enter image description here

Expected Output:
separate fields

_source: {
@timestamp: "2022-01-12T09:12:36.904Z",
"date": "28-09-2021",
 "language": " C++",
 "comment": 11,
  "code": 106026
input: {
type: "log"
},

Please suggest to decode the multiline json in filebeat?

2

How about using json options right after the multiline ones?

3

Hi @ChrsMark

i tried your suggestion, still no effect

filebeat.inputs:

  • type: log

    enabled: true

    paths:

    • /logs/*.json

    multiline.pattern: '^{'

    multiline.negate: true

    multiline.match: after

json.keys_under_root: true

json.add_error_key: true

json.message_key: "log"

processors:

  • decode_json_fields:

    fields: ["message"]

    process_array: false

    max_depth: "2"

    target: ""

    overwrite_keys: true

    add_error_key: false

output.elasticsearch:

Boolean flag to enable or disable the output module.

enabled: true

hosts: ["http://localhost:9200"]

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.