# Parse mix of String, JSON and KeyValue

**URL:** <https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791>\
**Category:** Logstash\
**Created:** [January 18, 2019, 12:42pm UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791 "2019-01-18T12:42:08Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mohammed\_Farhan\_Ejaz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammed_farhan_ejaz/32/80948_2.png) [@Mohammed\_Farhan\_Ejaz](https://discuss.elastic.co/u/Mohammed_Farhan_Ejaz)\
**Post date:** [January 18, 2019, 12:42pm UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791/1 "2019-01-18T12:42:08Z")

</div>

`2019-01-18T11:33:40,011 WXXX11100111L 43992 CM http-nio-8080-exec-8 MY-TEST POST configurations {a=b} {c=d, x=y} { "x": "abc", "country":"india", "lang":"EN", "project":"my-project"} 200 4266`

I have the above log entry to be stashed using grok filter. The intent is to extract {a=b} & {c=d, x=y} as key value pair which could be blank also {}, and the the next token as a json information from the content. A sample grok filter could be like this.

`%{TIMESTAMP_ISO8601:time} %{NOTSPACE:hostName} %{INT:pid} %{WORD:module} %{NOTSPACE:thread} %{NOTSPACE:CorrelationId} %{WORD:verb} %{NOTSPACE:path} %{???:kvQueryParameters} %{???:kvPathParameters} %{???:jsonObject} %{INT:response} %{INT:responseTime}`

Really struggling to think through how to define the patterns for these three tokens in the log ? Moreover the value in key value pairs could contain url encoded strings while the JSON string could be {} to complex object.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 18, 2019, 1:39pm UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791/2 "2019-01-18T13:39:26Z")

</div>

I would split that grok to parse all the easy bits, then remove them from the message.

```
grok {
    break_on_match => false
    match => {
        "message" => [
            "^%{TIMESTAMP_ISO8601:time} %{NOTSPACE:hostName} %{INT:pid} %{WORD:module} %{NOTSPACE:thread} %{NOTSPACE:CorrelationId} %{WORD:verb} %{NOTSPACE:path}",
            " %{INT:response} %{INT:responseTime}$"
        ]
    }
}
mutate {
    gsub => [
        "message", " [0-9]+ [0-9]+$", "",
        "message", "^[^]+ [^]+ [^]+ [^]+ [^]+ [^]+ [^]+ [^]+ ", ""
    ]
}

```

Now that you just have the key-value pairs and the json, then provided the key-value pairs do not contain } you can get those using

```
grok { match => { "message" => ["^{(?<kv1>[^}]*)}" ] } }
mutate { gsub => ["message", "^{[^}]*} ", "" ] }

```

You do that again for the second one and everything that is left is json.

---

<div class="post-metadata">

**Author:** ![satyajeet\_dutt](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@satyajeet\_dutt](https://discuss.elastic.co/u/satyajeet_dutt)\
**Post date:** [February 4, 2019, 5:49am UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791/3 "2019-02-04T05:49:59Z")

</div>

This is not working for me,the issue is with the path parameter.There is spaces in between and I am not able to remove those.it's like this {a=b, c=d, e=f} {???:jsonObject}.  
The output that I am receiving is like  
kvpathparameter=\> "{a=b,",  
jsonObject=\> "c=d, e=f} {???:jsonObject}"

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 4, 2019, 5:50am UTC](https://discuss.elastic.co/t/parse-mix-of-string-json-and-keyvalue/164791/4 "2019-03-04T05:50:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.