Parse mix of String, JSON and KeyValue

Elastic Stack Logstash
1

2019-01-18T11:33:40,011 WXXX11100111L 43992 CM http-nio-8080-exec-8 MY-TEST POST configurations {a=b} {c=d, x=y} { "x": "abc", "country":"india", "lang":"EN", "project":"my-project"} 200 4266

I have the above log entry to be stashed using grok filter. The intent is to extract {a=b} & {c=d, x=y} as key value pair which could be blank also {}, and the the next token as a json information from the content. A sample grok filter could be like this.

%{TIMESTAMP_ISO8601:time} %{NOTSPACE:hostName} %{INT:pid} %{WORD:module} %{NOTSPACE:thread} %{NOTSPACE:CorrelationId} %{WORD:verb} %{NOTSPACE:path} %{???:kvQueryParameters} %{???:kvPathParameters} %{???:jsonObject} %{INT:response} %{INT:responseTime}

Really struggling to think through how to define the patterns for these three tokens in the log ? Moreover the value in key value pairs could contain url encoded strings while the JSON string could be {} to complex object.

2

I would split that grok to parse all the easy bits, then remove them from the message.

grok {
    break_on_match => false
    match => {
        "message" => [
            "^%{TIMESTAMP_ISO8601:time} %{NOTSPACE:hostName} %{INT:pid} %{WORD:module} %{NOTSPACE:thread} %{NOTSPACE:CorrelationId} %{WORD:verb} %{NOTSPACE:path}",
            " %{INT:response} %{INT:responseTime}$"
        ]
    }
}
mutate {
    gsub => [
        "message", " [0-9]+ [0-9]+$", "",
        "message", "^[^ ]+ [^ ]+ [^ ]+ [^ ]+ [^ ]+ [^ ]+ [^ ]+ [^ ]+ ", ""
    ]
}

Now that you just have the key-value pairs and the json, then provided the key-value pairs do not contain } you can get those using

grok { match => { "message" => [ "^{(?<kv1>[^}]*)}" ] } }
mutate { gsub => [ "message", "^{[^}]*} ", "" ] }

You do that again for the second one and everything that is left is json.

3

This is not working for me,the issue is with the path parameter.There is spaces in between and I am not able to remove those.it's like this {a=b, c=d, e=f} {???:jsonObject}.
The output that I am receiving is like
kvpathparameter=> "{a=b,",
jsonObject=> "c=d, e=f} {???:jsonObject}"

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.