# Nested KV Filter in logstash **URL:** \ **Category:** Logstash\ **Created:** [May 2, 2018, 10:01am UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223 "2018-05-02T10:01:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![charan.gandra](https://avatars.discourse-cdn.com/v4/letter/c/a87d85/32.png) [@charan.gandra](https://discuss.elastic.co/u/charan.gandra)\ **Post date:** [May 2, 2018, 10:01am UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223/1 "2018-05-02T10:01:23Z")
Hi, Here is a line from my audit.log file "type=CRED\_DISP msg=audit(1525254633.039:542208): pid=35761 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam\_env,pam\_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success' I would like to parse the above line into key value pairs. Problem is the "msg" field which appears twice. I've used the basic KV filter and got the below output. { "message" =\> "type=CRED\_DISP msg=audit(1525254633.039:542208): pid=35761 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam\_env,pam\_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'", "@version" =\> "1", "host" =\> "localhost", "ses" =\> "4294967295", "path" =\> "/var/log/audit/audit.log", "type" =\> "CRED\_DISP", "auid" =\> "4294967295", "msg" =\> [ [0] "audit(1525254633.039:542208):", [1] "op=PAM:setcred grantors=pam\_env,pam\_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success" ], "pid" =\> "35761", "uid" =\> "0", "@timestamp" =\> 2018-05-02T09:51:25.626Z } Here is my logstash filter: input { file { path =\> "/var/log/audit/audit.log" start\_position =\> "beginning" } } filter { if [path] == "/var/log/audit/audit.log" { ``` kv { value_split => "=" } ``` } } output { if [path] == "/var/log/audit/audit.log" { stdout { codec =\> rubydebug } } }: I want output something like below: { "message" =\> "type=CRED\_DISP msg=audit(1525254633.039:542208): pid=35761 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:setcred grantors=pam\_env,pam\_fprintd acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=? res=success'", "@version" =\> "1", "host" =\> "localhost", "ses" =\> "4294967295", "path" =\> "/var/log/audit/audit.log", "type" =\> "CRED\_DISP", "auid" =\> "4294967295", "msg" =\> "audit(1525254633.039:542208):", "op" =\> PAM:setcred2 "grantors" =\> "pam\_env,pam\_fprintd" "acct =\> ""root"" "exe" =\> ""/usr/bin/sudo"" "hostname" =\> "?" "addr" =\> "?" "terminal" =\> "?" "res" =\> "success" ], "pid" =\> "35761", "uid" =\> "0", "@timestamp" =\> 2018-05-02T09:51:25.626Z } ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [May 3, 2018, 12:47am UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223/2 "2018-05-03T00:47:20Z")
There is a [`recursive`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html#plugins-filters-kv-recursive) directive on the KV Filter Plugin, but the duplication of `msg` is going to be problematic because the recursive parse will attempt to create sub-keys of `msg`, but that will conflict with the string key that doesn't have sub-keys. If it is _only_ the `msg` key that is duplicated, and one of those duplicates is _always_ single-quote nested kv, I think we can get what you're after by avoiding the collision before we pass it to the KV Filter Plugin: ```auto filter { mutate { gsub => ["message", " msg='", " _nested='"] } kv { source => "message" } kv { source => "_nested" remove_field => "_nested" } } ``` With the above and Logstash 6.2.2, I get the following; note that the KV filter will capture the _contents_ of quoted values (but not the quotes themselves): ```auto { "exe" => "/usr/bin/sudo", "pid" => "35761", "hostname" => "?", "op" => "PAM:setcred", "grantors" => "pam_env,pam_fprintd", "@timestamp" => 2018-05-03T00:45:05.782Z, "uid" => "0", "addr" => "?", "ses" => "4294967295", "acct" => "root", "msg" => "audit(1525254633.039:542208):", "terminal" => "?", "message" => "\"type=CRED_DISP msg=audit(1525254633.039:542208): pid=35761 uid=0 auid=4294967295 ses=4294967295 _nested='op=PAM:setcred grantors=pam_env,pam_fprintd acct=\"root\" exe=\"/usr/bin/sudo\" hostname=? addr=? terminal=? res=success'", "res" => "success", "@version" => "1", "auid" => "4294967295", "host" => "castrovel.local", "\"type" => "CRED_DISP" } ``` ---
**Author:** ![charan.gandra](https://avatars.discourse-cdn.com/v4/letter/c/a87d85/32.png) [@charan.gandra](https://discuss.elastic.co/u/charan.gandra)\ **Post date:** [May 3, 2018, 11:10am UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223/3 "2018-05-03T11:10:27Z")
Thank you for your help, its working as I expected. ---
**Author:** ![imfassistance](https://avatars.discourse-cdn.com/v4/letter/i/919ad9/32.png) [@imfassistance](https://discuss.elastic.co/u/imfassistance)\ **Post date:** [May 11, 2018, 3:30pm UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223/4 "2018-05-11T15:30:30Z")
Thanks for sharing, very helpful [CMC Vellore](https://treatgo.com/hospital/christian-medical-college-vellore/) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2018, 3:30pm UTC](https://discuss.elastic.co/t/nested-kv-filter-in-logstash/130223/5 "2018-06-08T15:30:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.