Logstash kv filter extract json array issue

[cheriemilk]

Hi,

I have below json file, and the json object contains array. I want to use kv filter ( not json filter) to extract corresponding kv paris,
{
"DP": "dbpool1",
"CMID": "TALSCMK",
"CMN": "HuaWei",
"UID": "lokamoto1",
"UN": "lokamoto",
"PUID": "admin1",
"UL": "zh_CN",
"CIP": "192.168.56.3",
"SN": "qacandrot_TALSCMK",
"DC": "DC08",
"CLN": "TalentSearchController",
"MID": "SCM",
"PID": "TalentSearch",
"PQ": "v11",
"AC": "TalentSearch",
"SCM.TS.TS.IIN": "true",
"SCM.TS.TS.MACO": "false",
"SCM.TS.TS.COND": ["KC", "BC", "PC", "FC", "RC"],
** "SCM.TS.TS.BC": ["age", "fax", "ethnicity"],**
** "SCM.TS.TS.PC": ["achievements", "languages"],**
** "SCM.TS.TS.FC": ["department", "location"],**
** "SCM.TS.TS.RC": ["sysOverallPotential", "sysOverallCustom1"],**
"SCM.TS.TS.NR": 200
}

My kv pair configured

kv {
source => "message"
allow_duplicate_values => true
field_split => ","
value_split => ":"

   }

extracted information after kv filter
""CMN"" => "HuaWei",
""UN"" => "lokamoto",
""PUID"" => "admin1",
""AC"" => "TalentSearch",
""SCM.TS.TS.IIN"" => "true",
""SCM.TS.TS.RC"" => ""sysOverallPotential","sysOverallCustom1"",
""SCM.TS.TS.PC"" => ""achievements","languages"",
""CMID"" => "TALSCMK",
""SCM.TS.TS.BC"" => ""age","fax","ethnicity"",
""SN"" => "qacandrot_TALSCMK",
""SCM.TS.TS.COND"" => ""KC","BC","PC","FC","RC"",
""PID"" => "TalentSearch",
"@version" => "1",
"host" => "PVGN50859047A",
""SCM.TS.TS.NR"" => "200}\r",
""SCM.TS.TS.FC"" => ""department","location"",
""PQ"" => "v11",
""DC"" => "DC08",
"path" => "C:\elkstack\elasticsearch-6.5.1\logs\kv.log",
""MID"" => "SCM",
""UL"" => "zh_CN",
""CIP"" => "192.168.56.3",
"{"DP"" => "dbpool1",
""CLN"" => "TalentSearchController",
"@timestamp" => 2019-01-11T02:50:04.029Z,
""SCM.TS.TS.MACO"" => "false",
""UID"" => "lokamoto1"

In Kibana, the display looks like below after kv filter, and when doing aggregation based on key whose value is array. For example, CM.TS.TS.COND, kibana treats corresponding value "KC", "BC", "PC", "FC", "RC" as string, instead of array.

Expected Chart I want get is to aggregate based on the values in the array.

How should I configure kv filter to let kibana aggreate the chart based on arrays?

[Christian_Dahlqvist]

This is what the JSON filter was designed for, why not use it?

[cheriemilk]

I am just seeking for a kv filter solution. (json filter does work)

[cheriemilk]

Any idea how to make it by kv filter or a combination filters?

[Christian_Dahlqvist]

No, I do not know. As there is a filter specifically designed for parsing this I also do not see the point in this exercise, so will unfortunately not be able to help.

[cheriemilk]

Hi Christian,

I think I found the solution. Configure below filters with KV+Ruby returns my expected behavior in Kibana. the pie chart can be aggrated based on array elements successfully.

    kv {
        source => "message"
        field_split => ","
        value_split => ":"
       }
   
    ruby {
          code => " 
	           event.to_hash.each { |k,v| 
				                            if v.is_a?(String)&&v.length>2
                                                    event.set(k, v.split(',')) 
											else
												    event.set(k, v)
                                            end 
										    }"
         }

[cheriemilk]

AND more easiler way is the json filter, either specify it in input, filter or output.

1. configure in input
   input{

file{
path => "C:\elkstack\elasticsearch-6.5.1\logs\kv.log"
start_position => "beginning"
sincedb_path => "null"
codec => "json"
}

}

2. configure in filter
   filter {
   json {
   source => "message"
   }
   }

3. configure in output

stdout {
codec => "json"
}