# Parsing a mix of String and JSON fields with the kv() filter

**URL:** <https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093>\
**Category:** Logstash\
**Created:** [July 6, 2017, 12:11pm UTC](https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093 "2017-07-06T12:11:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![olivier1](https://avatars.discourse-cdn.com/v4/letter/o/85f322/32.png) [@olivier1](https://discuss.elastic.co/u/olivier1)\
**Post date:** [July 6, 2017, 12:11pm UTC](https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093/1 "2017-07-06T12:11:28Z")

</div>

Hi,

I'm trying to do something that looks fairly simple according to the doc: trimming start and end double quotes from a field using the kv{} filter. I've tried a whole bunch of combinations like '"', """, '"' but none is working. Tried also with the mutate{} filter, but no luck.

Here's a what a log line looks like:  
`2017-07-06 11:00:00,154 UTC, refId="105ad5d7-7dc0-4379-9d0c-256b56795467", body="{"quantity":"2","startTimes":["2017-07-06T11:00:00+0000","2017-07-07T11:15:00+0000"]}"`

The test configuration I am using:

```
input {
    stdin {}
}

filter {
grok {
    match => ["message", "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:rest}"]
    remove_field => "message"
}

date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
    target => "@timestamp"
    remove_field => "timestamp"
}

kv {
    source => "rest" 
    remove_field => ["rest"]
    trim_value => "\""
}

# json {  
# remove_field => "body"
# source => "body"
# target => "payload"
# }
}   

output {
# elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

```

And the output I got:  
{  
"@timestamp" =\> 2017-07-06T11:00:00.154Z,  
"@version" =\> "1",  
"host" =\> "QL000000026",  
"refId" =\> "105ad5d7-7dc0-4379-9d0c-256b56795467",  
"body" =\> "{"  
}

So the body is truncated, and if I remove the double quotes manually, it gives me what I expect:  
{  
"@timestamp" =\> 2017-07-06T11:00:00.154Z,  
"@version" =\> "1",  
"host" =\> "QL000000026",  
"refId" =\> "105ad5d7-7dc0-4379-9d0c-256b56795467",  
"body" =\> "{"quantity":"2","startTimes":["2017-07-06T11:00:00+0000","2017-07-07T11:15:00+0000"]}\r"  
}

Any ideas what I am doing wrong? Or maybe there is a better way to Grok this in the first place?

TIA,  
Olivier.

---

<div class="post-metadata">

**Author:** ![olivier1](https://avatars.discourse-cdn.com/v4/letter/o/85f322/32.png) [@olivier1](https://discuss.elastic.co/u/olivier1)\
**Post date:** [July 11, 2017, 9:08am UTC](https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093/2 "2017-07-11T09:08:33Z")

</div>

Hi all,

I did more tests and I believe the problem could be with the field/value splitter. I tried a few other things but this still gives me a truncated body field. Still not clear why when I remove the double quotes this get parsed as expected...

Is there a way I could extract this field with something else than kv() filter and feed it directly to JSON filter?

My latest configuration:

```
input {
    stdin {}
}

filter {
    grok {
        pattern_definitions => {
            DATESTAMP_WITH_TZ => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{TZ}"
        }
        match => ["message", "%{DATESTAMP_WITH_TZ:timestamp}, %{GREEDYDATA:rest}"]
        remove_field => "message"
    }

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS ZZZ"]
        target => "@timestamp"
        remove_field => "timestamp"
    }

    kv {
        source => "rest" 
        remove_field => ["rest"]
        field_split => "[,\s]"
        value_split => "="
        trim_key => "\s"
        trim_value => "\s"
        # exclude_keys => ["body"]
    }

    # json {  
    # remove_field => "body"
    # source => "body"
    # target => "payload"
    # }
}   

output {
# elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![olivier1](https://avatars.discourse-cdn.com/v4/letter/o/85f322/32.png) [@olivier1](https://discuss.elastic.co/u/olivier1)\
**Post date:** [July 11, 2017, 5:29pm UTC](https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093/3 "2017-07-11T17:29:05Z")

</div>

Alright, after some trial/errors, I found a workaround for now: not sure how/why it works but replacing the double quotes with simple quotes make kv() happy with the value of the field.

So here's what I am using now:  
input {  
stdin {}  
}

```
filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:rest}"]
        remove_field => "message"
    }

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
        target => "@timestamp"
        remove_field => "timestamp"
    }

    mutate {
        gsub => [
                "rest", '"{', "'{",
                "rest", '}"', "}'"
            ]
    }

    kv {
        source => "rest" 
        remove_field => ["rest"]
        field_split => ", "
        value_split => "="
        trim_key => " "
        trim_value => " "

    }

    json {  
        remove_field => "body"
        source => "body"
        target => "payload"
    }
}

output {
# elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 8, 2017, 5:29pm UTC](https://discuss.elastic.co/t/parsing-a-mix-of-string-and-json-fields-with-the-kv-filter/92093/4 "2017-08-08T17:29:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.