Parse array in Json log file with Filebeat 7.7.1

[roshann (roshann agarwal)]

Hi,
I have a json log file as below:

{ "Format": "IDEA0", "ID": "1c5ae2e1-bf16-43d6-9233-5865f83ad180", "DetectTime": "2022-12-03T11:17:23.589015+00:00", "EventTime": "2022-12-03T11:17:23.589020+00:00", "Category": ["Anomaly.Connection"], "Confidence": 0.8, "Source": [{"IP4": ["192.168.2.16"], "Type": ["Malware"]}], "Target": [{"IP4": ["40.190.70.193"], "Type": ["Malware"]}], "Attach": [{"Content": "a connection without DNS resolution to IP: 40.190.70.193 AS: DXTL Tseung Kwan O Service", "ContentType": "text/plain"}] }
{ "Format": "IDEA0", "ID": "b110759c-3f16-47f5-8661-04021e547dd0", "DetectTime": "2022-12-03T11:17:23.598444+00:00", "EventTime": "2022-12-03T11:17:23.598450+00:00", "Category": ["Anomaly.Connection"], "Confidence": 0.8, "Source": [{"IP4": ["192.168.2.16"], "Type": ["Malware"]}], "Target": [{"IP4": ["40.60.70.206"], "Type": ["Malware"]}], "Attach": [{"Content": "a connection without DNS resolution to IP: 40.60.70.206 AS: Korea Telecom", "ContentType": "text/plain"}] }

I want to send the logs using Filebeat only version 7.7.1; however the "decode_json_fields" processor is being able to decode fields but not the array inside the json; specially I am looking for the fields to decode "Source" and "Target" along with the sub fields "IP4", "Type"; it is decoding the field as a single target with the sub fields inside as below:

Source
{
"Type": [
"Malware"
],
"IP4": [
"192.168.2.16"
]
}

The result should look like as below:

Source.IP4 192.168.56.105
Source.Type MultipleUserAgent

Please help I need this very urgent as i missed the deadline
The file beat config file is as below:

filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /var/log/sips/alerts.json
    #json.keys_under_root: false

processors:

• decode_json_fields:
  fields: ["message"]
  process_array: true
  max_depth: 10
  target: ""
  overwrite_keys: true
  add_error_key: true
  #the below is not working even though it is there
• extract_array:
  field: Source
  mappings:
  Source.IP4: 0
  Source.Type: 1
  Source.Port: 2
  Source.Proto: 3
  overwrite_keys: true

Thanks in advance for ur efforts

[ashishtiwari1993 (Ashish Tiwari)]

Hi Roshann,

Here is something which worked for me:

filebeat.yml

- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /tmp/fb_test.log
  parsers:
    - ndjson:
        target: ""

I am using ndjson parser here.

Adding Sample json line

echo '{"a":"b","my_array":["ele1","ele2"]}' >> /tmp/fb_test.log

Similarly tried with your json

echo '{ "Format": "IDEA0", "ID": "1c5ae2e1-bf16-43d6-9233-5865f83ad180", "DetectTime": "2022-12-03T11:17:23.589015+00:00", "EventTime": "2022-12-03T11:17:23.589020+00:00", "Category": ["Anomaly.Connection"], "Confidence": 0.8, "Source": [{"IP4": ["192.168.2.16"], "Type": ["Malware"]}], "Target": [{"IP4": ["40.190.70.193"], "Type": ["Malware"]}], "Attach": [{"Content": "a connection without DNS resolution to IP: 40.190.70.193 AS: DXTL Tseung Kwan O Service", "ContentType": "text/plain"}] }
{ "Format": "IDEA0", "ID": "b110759c-3f16-47f5-8661-04021e547dd0", "DetectTime": "2022-12-03T11:17:23.598444+00:00", "EventTime": "2022-12-03T11:17:23.598450+00:00", "Category": ["Anomaly.Connection"], "Confidence": 0.8, "Source": [{"IP4": ["192.168.2.16"], "Type": ["Malware"]}], "Target": [{"IP4": ["40.60.70.206"], "Type": ["Malware"]}], "Attach": [{"Content": "a connection without DNS resolution to IP: 40.60.70.206 AS: Korea Telecom", "ContentType": "text/plain"}] }' >> /tmp/fb_test.log

Its inserted in proper format.

Though i have tested on Version 8.5.3 But parser is available from version 7.13

[roshann (roshann agarwal)]

Thanks for the reply but i need it a fix for version 7.7.1, i guess filestream is not there..
Below is the error:

Exiting: Error while initializing input: Error creating input. No such input type exist: 'filestream'