Parsing a mix of String and JSON fields with the kv() filter

[olivier1]

Hi,

I'm trying to do something that looks fairly simple according to the doc: trimming start and end double quotes from a field using the kv{} filter. I've tried a whole bunch of combinations like '"', """, '"' but none is working. Tried also with the mutate{} filter, but no luck.

Here's a what a log line looks like:
2017-07-06 11:00:00,154 UTC, refId="105ad5d7-7dc0-4379-9d0c-256b56795467", body="{"quantity":"2","startTimes":["2017-07-06T11:00:00+0000","2017-07-07T11:15:00+0000"]}"

The test configuration I am using:

input {
    stdin {}
}

filter {
grok {
    match => ["message", "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:rest}"]
    remove_field => "message"
}

date {
    match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
    target => "@timestamp"
    remove_field => "timestamp"
}

kv {
    source => "rest" 
    remove_field => ["rest"]
    trim_value => "\""
}

# json {  
#     remove_field => "body"
#     source => "body"
#     target => "payload"
# }
}   

output {
#  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

And the output I got:
{
"@timestamp" => 2017-07-06T11:00:00.154Z,
"@version" => "1",
"host" => "QL000000026",
"refId" => "105ad5d7-7dc0-4379-9d0c-256b56795467",
"body" => "{"
}

So the body is truncated, and if I remove the double quotes manually, it gives me what I expect:
{
"@timestamp" => 2017-07-06T11:00:00.154Z,
"@version" => "1",
"host" => "QL000000026",
"refId" => "105ad5d7-7dc0-4379-9d0c-256b56795467",
"body" => "{"quantity":"2","startTimes":["2017-07-06T11:00:00+0000","2017-07-07T11:15:00+0000"]}\r"
}

Any ideas what I am doing wrong? Or maybe there is a better way to Grok this in the first place?

TIA,
Olivier.

[olivier1]

Hi all,

I did more tests and I believe the problem could be with the field/value splitter. I tried a few other things but this still gives me a truncated body field. Still not clear why when I remove the double quotes this get parsed as expected...

Is there a way I could extract this field with something else than kv() filter and feed it directly to JSON filter?

My latest configuration:

input {
    stdin {}
}

filter {
    grok {
        pattern_definitions => {
            DATESTAMP_WITH_TZ => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{TZ}"
        }
        match => ["message", "%{DATESTAMP_WITH_TZ:timestamp}, %{GREEDYDATA:rest}"]
        remove_field => "message"
    }

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS ZZZ"]
        target => "@timestamp"
        remove_field => "timestamp"
    }

    kv {
        source => "rest" 
        remove_field => ["rest"]
        field_split => "[,\s]"
        value_split => "="
        trim_key => "\s"
        trim_value => "\s"
        # exclude_keys => ["body"]
    }

    # json {  
    #     remove_field => "body"
    #     source => "body"
    #     target => "payload"
    # }
}   

output {
#  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

[olivier1]

Alright, after some trial/errors, I found a workaround for now: not sure how/why it works but replacing the double quotes with simple quotes make kv() happy with the value of the field.

So here's what I am using now:
input {
stdin {}
}

filter {
    grok {
        match => ["message", "%{TIMESTAMP_ISO8601:timestamp}%{GREEDYDATA:rest}"]
        remove_field => "message"
    }

    date {
        match => ["timestamp", "yyyy-MM-dd HH:mm:ss,SSS", "yyyy-MM-dd HH:mm:ss", "ISO8601"]
        target => "@timestamp"
        remove_field => "timestamp"
    }

    mutate {
        gsub => [
                "rest", '"{', "'{",
                "rest", '}"', "}'"
            ]
    }

    kv {
        source => "rest" 
        remove_field => ["rest"]
        field_split => ", "
        value_split => "="
        trim_key => " "
        trim_value => " "

    }

    json {  
        remove_field => "body"
        source => "body"
        target => "payload"
    }
}

output {
#  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}