# How to create grok/json filter to parse the below json format

**URL:** <https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022>\
**Category:** Logstash\
**Created:** [February 2, 2022, 3:24am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022 "2022-02-02T03:24:45Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 3:24am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/1 "2022-02-02T03:24:45Z")

</div>

Hi Guys,

I want to parse this JSON to Kibana using Logstash

```auto
{
"Format": "IDEA0",
"ID": "2b03eb1f-fc4c-4f67-94e5-31c9fb32dccc",
"DetectTime": "2022-01-31T08:16:12.600470+07:00",
"EventTime": "2022-01-31T01:23:01.637438+00:00",
"Category": ['Intrusion.Botnet'],
"Confidence": 0.03,
"Note": "C&C channel, destination IP: 192.168.1.24 port: 8007/tcp score: 0.9324",
"Source": [{'IP4': ['192.168.1.25'], 'Type': ['CC']}]
}

```

I want that **ID, Detect Time, Event Time, Category, Confidence, Note, Source** is a single field so later i can do visualization in kibana.

Here's what I'm already trying to do

```auto
input {
        file {
                path => "/home/ubuntu/Downloads/StratosphereLinuxIPS/output/*.json"
                start_position => "beginning"
                sincedb_path => "/dev/null"
        }
}

filter {
        json {
                source => "message"
        }
}

output {
        elasticsearch {
                hosts => ["localhost:9200"]
                index => "test-test"
                user => "***"
                password => "***"
        }
        stdout{}
}

```

But the field is not separated correctly

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5efee4d9f62679c189f7f6f472653c8cd0c2cf94.png)

Thanks.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 2, 2022, 4:16am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/2 "2022-02-02T04:16:05Z")

</div>

Add multiline codec to input plugin

```auto
input {
        file {
                path => "/home/ubuntu/Downloads/StratosphereLinuxIPS/output/*.json"
                start_position => "beginning"
                sincedb_path => "/dev/null"
                codec = > multiline { pattern = > "^{$" negate = > "true" what = > "previous" }
        }
}

```

Your input plugin generates events for each line.

---

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 4:34am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/3 "2022-02-02T04:34:40Z")

</div>

Hi Tomo,

I've tried the code you give me, but still, output still gives me un-separated fields.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 2, 2022, 4:45am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/4 "2022-02-02T04:45:38Z")

</div>

Does multiline codec make any change at all?  
What is your current output if you use:

```auto
output {
   stdout { codec => rubydebug }
}

```

---

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 5:41am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/5 "2022-02-02T05:41:01Z")

</div>

oh my bad, there's a change in the output

 ![Screenshot 2022-02-02 124016](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1dfa8845779c009ac72f9fd1c3c2e70dd51bae85.png)

all the JSON are contained in the one message fields.

Note: the rubydebug give the same output

---

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 5:45am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/6 "2022-02-02T05:45:25Z")

</div>

just to make sure what i mean in this sentence

" I want that **ID, Detect Time, Event Time, Category, Confidence, Note, Source** is a single field so later i can do visualization in kibana."

is not all those fields in 1 field, I want those field are have own fields.

So there's a fields called ID, Detect Time, Event Time, etc

Thanks.

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 2, 2022, 6:03am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/7 "2022-02-02T06:03:43Z")

</div>

> [@Adabi\_Raihan](#):
>
> to make sure what i mean in this sentence

I suppose I have understood your intent.

You can get JSON from Discover. Please share documentw by texts because sharing by screenshot is less informative in most cases.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/f/1fe85c7ebebe65e396edf4af91bf47168f0523fe.png)

As there are `_jsonparsefailure` in tags, the cause of the problem can be identified as the JSON filter plugin.

Single quotation is not consistent with the original JSON format. Try:

```auto
filter {
        mutate => {
                gsub => ["message", "'","\""]
        }
        json {
                source => "message"
                # remove_field => "message" #if you don't need this field
        }
}

```

---

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 6:20am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/9 "2022-02-02T06:20:32Z")

</div>

still contained in 1 fields

```auto
@timestamp:
    Feb 2, 2022 @ 13:17:45.986
@version:
    1
host:
    ubuntu2004
message:
    { "Format": "IDEA0", "ID": "5031c428-dae4-4c47-81b1-c672f1e3325f", "DetectTime": "2022-01-31T11:04:05.206553+07:00", "EventTime": "2022-01-31T04:06:40.906870+00:00", "Category": [\"Anomaly.Connection\"], "Confidence": 0.8, "Note": "a connection without DNS resolution to IP: 34.117.59.81. ", "Source": [{\"IP4\": [\"192.168.1.24\"], \"Type\": [\"Malware\"]}], "Target": [{\"IP4\": [\"34.117.59.81\"], \"Type\": [\"Malware\"]}] }
path:
    /home/ubuntu/Downloads/StratosphereLinuxIPS/output/alerts.json
tags:
    multiline, _jsonparsefailure
_id:
    hR0UuX4Bi0YbDUJ9-J-c
_index:
    test-keempat
_score:
    - 
_type:
    _doc 

```

Here's the screenshot (if needed)

 ![4](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bfbbfa0e665bf49eeb491def3ac024cf14a2e337.png)

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [February 2, 2022, 7:22am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/10 "2022-02-02T07:22:36Z")

</div>

Turn on `config.support_escapes` setting in [logstash.yml](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html)

Or try

```auto
mutate => {
                gsub => ["message", "'",'"']
        }

```

---

<div class="post-metadata">

**Author:** ![Adabi\_Raihan](https://avatars.discourse-cdn.com/v4/letter/a/67e7ee/32.png) [@Adabi\_Raihan](https://discuss.elastic.co/u/Adabi_Raihan)\
**Post date:** [February 2, 2022, 8:49am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/11 "2022-02-02T08:49:28Z")

</div>

Thank You Tomo, work like a charm 🙂

I'm using this one

> [@Tomo\_M](#):
>
> ```auto
> mutate => {
> gsub => ["message", "'",'"']
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2022, 8:50am UTC](https://discuss.elastic.co/t/how-to-create-grok-json-filter-to-parse-the-below-json-format/296022/12 "2022-03-02T08:50:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.