Problem processing multiline json

Elastic Stack Beats
1

MY INPUT FILE :

[
    {
        "value":  "drrrrrrrrrrrrropfields_NTNT10",
        "Id":  "drrrrrrrrrropfields_10",
        "PSShowComputerName":  "dropfields_10"
    },
	{
        "value":  "dropfields_nT8",
        "PSN":  "dropfields_8",
        "Id":  "dropfields_8",
        "PSShowComputerName":  "dropfields_8"
    },
	{
        "value":  "dropfields_NT9",
        "PSN":  "dropfields_9",
        "Id":  "dropfields_9",
        "PSShowComputerName":  "dropfields_9"
    }
	
]

MY YML FILE :

filebeat.inputs:
      - type: log
        enabled: true
        paths:
          - "C:\\Program Files\\Filebeat\\data\\new\\*.json"
        scan_frequency: 1s
        tail_files: false
        multiline.pattern: '},'
        multiline.negate: true
        multiline.match: before
processors:
      - drop_fields:
          fields: ["ecs.version", "host.name","agent.id","agent.hostname","agent.version","agent.type","agent.ephemeral_id","agent","ecs","host","log.file.path","log.file","log.flags","log.offset","log","input.type","input","@metadata.version"]
      - decode_json_fields:
          fields: ["message"]
          target: "datas"
output.console:
        pretty: true

MY OUTPUT


{
  "@timestamp": "2020-09-30T05:01:02.650Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0"
  },
  "message": "[\n    {\n        \"value\":  \"drrrrrrrrrrrrropfields_NTNT10\",\n        \"Id\":  \"drrrrrrrrrropfields_10\",\n        \"PSShowComputerName\":  \"dropfields_
10\"\n    },"
}
{
  "@timestamp": "2020-09-30T05:01:02.650Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0"
  },
  "message": "\t{\n        \"value\":  \"dropfields_nT8\",\n        \"PSN\":  \"dropfields_8\",\n        \"Id\":  \"dropfields_8\",\n        \"PSShowComputerName\":  \"drop
fields_8\"\n    },"
}
{
  "@timestamp": "2020-09-30T05:01:02.652Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.3.0"
  },
  "message": "\t{\n        \"value\":  \"dropfields_NT9\",\n        \"PSN\":  \"dropfields_9\",\n        \"Id\":  \"dropfields_9\",\n        \"PSShowComputerName\":  \"drop
fields_9\"\n    }\n\t",
  "datas": {
    "value": "dropfields_NT9",
    "PSN": "dropfields_9",
    "Id": "dropfields_9",
    "PSShowComputerName": "dropfields_9"
  }
}

As you can see only the last event of my JSON file is decoded properly as the first two have unwanted characters before and after the curly parenthesis "{" and "}" .

how do I fix this??

2

Hello @niv,

are you using Logstash?
this issue can be resolved in Logstash using the Json filter.

or target to message in the JSON Processor


Cheers

3

hey as you can see I am using a decode_json_fields. But the problem is that there are unwanted characters get added to the first two json events when I use the multiline to parse it. my question is how do I avoid it ?

4

Can someone help please?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.