How to create grok/json filter to parse the below json format

[Adabi_Raihan]

Hi Guys,

I want to parse this JSON to Kibana using Logstash

{
"Format": "IDEA0",
"ID": "2b03eb1f-fc4c-4f67-94e5-31c9fb32dccc",
"DetectTime": "2022-01-31T08:16:12.600470+07:00",
"EventTime": "2022-01-31T01:23:01.637438+00:00",
"Category": ['Intrusion.Botnet'],
"Confidence": 0.03,
"Note": "C&C channel, destination IP: 192.168.1.24 port: 8007/tcp score: 0.9324",
"Source": [{'IP4': ['192.168.1.25'], 'Type': ['CC']}]
}

I want that ID, Detect Time, Event Time, Category, Confidence, Note, Source is a single field so later i can do visualization in kibana.

Here's what I'm already trying to do

input {
        file {
                path => "/home/ubuntu/Downloads/StratosphereLinuxIPS/output/*.json"
                start_position => "beginning"
                sincedb_path => "/dev/null"
        }
}

filter {
        json {
                source => "message"
        }
}

output {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "test-test"
                user => "***"
                password => "***"
        }
        stdout{}
}

But the field is not separated correctly

Thanks.

[Tomo_M (Tomohiro Mitani)]

Add multiline codec to input plugin

input {
        file {
                path => "/home/ubuntu/Downloads/StratosphereLinuxIPS/output/*.json"
                start_position => "beginning"
                sincedb_path => "/dev/null"
                codec = > multiline { pattern = > "^{$" negate = > "true" what = > "previous" }
        }
}

Your input plugin generates events for each line.

[Adabi_Raihan]

Hi Tomo,

I've tried the code you give me, but still, output still gives me un-separated fields.

[Tomo_M (Tomohiro Mitani)]

Does multiline codec make any change at all?
What is your current output if you use:

output {
   stdout { codec => rubydebug }
}

[Adabi_Raihan]

oh my bad, there's a change in the output

all the JSON are contained in the one message fields.

Note: the rubydebug give the same output

[Adabi_Raihan]

just to make sure what i mean in this sentence

" I want that ID, Detect Time, Event Time, Category, Confidence, Note, Source is a single field so later i can do visualization in kibana."

is not all those fields in 1 field, I want those field are have own fields.

So there's a fields called ID, Detect Time, Event Time, etc

Thanks.

[Tomo_M (Tomohiro Mitani)]

@Adabi_Raihan

to make sure what i mean in this sentence

I suppose I have understood your intent.

You can get JSON from Discover. Please share documentw by texts because sharing by screenshot is less informative in most cases.

As there are _jsonparsefailure in tags, the cause of the problem can be identified as the JSON filter plugin.

Single quotation is not consistent with the original JSON format. Try:

filter {
        mutate => {
                gsub => ["message", "'","\""]
        }
        json {
                source => "message"
                # remove_field => "message" #if you don't need this field
        }
}

[Adabi_Raihan]

still contained in 1 fields

@timestamp:
    Feb 2, 2022 @ 13:17:45.986
@version:
    1
host:
    ubuntu2004
message:
    { "Format": "IDEA0", "ID": "5031c428-dae4-4c47-81b1-c672f1e3325f", "DetectTime": "2022-01-31T11:04:05.206553+07:00", "EventTime": "2022-01-31T04:06:40.906870+00:00", "Category": [\"Anomaly.Connection\"], "Confidence": 0.8, "Note": "a connection without DNS resolution to IP: 34.117.59.81. ", "Source": [{\"IP4\": [\"192.168.1.24\"], \"Type\": [\"Malware\"]}], "Target": [{\"IP4\": [\"34.117.59.81\"], \"Type\": [\"Malware\"]}] }
path:
    /home/ubuntu/Downloads/StratosphereLinuxIPS/output/alerts.json
tags:
    multiline, _jsonparsefailure
_id:
    hR0UuX4Bi0YbDUJ9-J-c
_index:
    test-keempat
_score:
    - 
_type:
    _doc 

Here's the screenshot (if needed)

[Tomo_M (Tomohiro Mitani)]

Turn on config.support_escapes setting in logstash.yml

Or try

mutate => {
                gsub => ["message", "'",'"']
        }

[Adabi_Raihan]

Thank You Tomo, work like a charm

I'm using this one

@Tomo_M

mutate => {
                gsub => ["message", "'",'"']
        }