# Using KV and Grok to parse complex data

**URL:** <https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769>\
**Category:** Logstash\
**Created:** [June 2, 2021, 2:34pm UTC](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769 "2021-06-02T14:34:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![blackberrySherbet](https://avatars.discourse-cdn.com/v4/letter/b/7cd45c/32.png) [@blackberrySherbet](https://discuss.elastic.co/u/blackberrySherbet)\
**Post date:** [June 2, 2021, 2:34pm UTC](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769/1 "2021-06-02T14:34:55Z")

</div>

Hi,

My data is structured like this-

INFO {"datetime": "2021-06-1 22:13:29.469000", data:{"val1":3.14, "val2": 2.17}}  
INFO {"datetime": "2021-06-1 21:14:00.469000", data:{"val3":9}}

I want to use logstash to process this data to upload to elastic search.

I have 2 issues-

1. The key inside the 'data' parameter is dynamic. I think I can use KV to fix this. Am I correct?
2. The number of key-value pairs inside the data parameter can be just 1 or 10. IE, it is dynamic. How do I work with this? Do I nest grok instructions?

I have tried several times and failed to create a filter. Any help will be highly appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 2, 2021, 3:07pm UTC](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769/2 "2021-06-02T15:07:37Z")

</div>

Your log is not complex, it looks like a string followed by a json document, but it is not a valid json because the `data` is not inside double quotes, is this a typo or is this how it looks in your message?

Considering that this is how it looks, you can transform it in a valid json and use dissect and the json filter to parse it.

For example, using this as sample messages:

```auto
INFO {"datetime": "2021-06-1 22:13:29.469000", data:{"val1":3.14, "val2": 2.17}}
INFO {"datetime": "2021-06-1 21:14:00.469000", data:{"val3":9}}

```

The following filter will parse those messages:

```auto
filter {
    mutate {
        gsub => ["message", 'data:', '"data":']
    }
    dissect {
        mapping => {
            "message" => "%{logLevel} %{jsonData}"
        }
    }
    json {
        source => "jsonData"
    }
}

```

The `mutate` filter with `gsub` will transform the json part in a valid json, that can be parsed with the `json` filter.

The `dissect` filter will split your message in two, the first parte will have the log level information, in this case `INFO`, and the second will have the json data.

The `json` filter will parse your json, so the end result will be something like this:

```auto
{
  "logLevel": "INFO",
  "@timestamp": "2021-06-02T15:06:34.822Z",
  "datetime": "2021-06-1 22:13:29.469000",
  "data": {
    "val1": 3.14,
    "val2": 2.17
  },
  "jsonData": "{\"datetime\": \"2021-06-1 22:13:29.469000\", \"data\":{\"val1\":3.14, \"val2\": 2.17}}",
  "message": "INFO {\"datetime\": \"2021-06-1 22:13:29.469000\", \"data\":{\"val1\":3.14, \"val2\": 2.17}}"
}
{
  "logLevel": "INFO",
  "@timestamp": "2021-06-02T15:06:34.883Z",
  "datetime": "2021-06-1 21:14:00.469000",
  "data": {
    "val3": 9
  },
  "jsonData": "{\"datetime\": \"2021-06-1 21:14:00.469000\", \"data\":{\"val3\":9}}",
  "message": "INFO {\"datetime\": \"2021-06-1 21:14:00.469000\", \"data\":{\"val3\":9}}"
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2021, 3:08pm UTC](https://discuss.elastic.co/t/using-kv-and-grok-to-parse-complex-data/274769/3 "2021-06-30T15:08:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.