Create KV-pairs for field not in key:value format and use ruby filter on KV pairs

[parinitha.nagaraja]

Hello,

I am trying to apply KV filter to a field that does not have "key:value key:value" format. The format is "value(key) value(key) ...". Below are the details.

• 2021-06-28 19:06:15.848 [0000-XYZ] [appLog ] partition 100 in 4 buckets: [ 10(0) 20(1) 20(0) 50(1) ]

• I pass the log line through a grok filer and create a filed called [bucket_data].
  [bucket_data]: 10(0) 20(1) 20(0) 50(1)

• Goal : To create fields as below :
  bucket.0 : 30 (which is the sum of all values in bucket 0 i.e 10+20=30)
  bucket.1 : 70 (which is the sum of all values in bucket 1 i.e 20+50=70)

• How I think we can achieve this is by passing the [bucket_data] into a KV filter and then to a ruby filter to compute the sum.

Questions:

1. How do I use the KV filter in this case ? The [bucket_data] format is value(key) value(key) .... What should be the field_split, value_split, trim_key, trim_value ?

2. How to I apply ruby filter after this ? Please see the code below.

3. Is there a better approach? Please let me know.

if [bucket_data] {		
			kv {
				source => "[bucket_data]"
				field_split => ""
				value_split => ""
				trim_key => "\s"
				trim_value => "\s"			
				target => "[bucket]"
			}

			if [bucket] {
				ruby { 
					code => " 
						kv = event.get('[bucket]')					 
						kv.to_hash.each  { |key,values|	
							sum = 0
							values.each { |val| sum+=val };	
							event.set('[bucket][' + key + ']', sum.to_f);						
						}
					"
				}
			}								
		}

[Badger]

@parinitha.nagaraja

How do I use the KV filter in this case ?

I would not use a kv filter for this. There are two values that have a key "20", so sometimes the value is an array. Working around that will complicate things too much. I would just do it in ruby

    grok { match => { "message" => "\[(?<bucket_data>[^]]+)\]$" } }
    ruby {
        code => '
            data = event.get("bucket_data")
            if data
                matches = data.scan(/(\d+)\((\d)\)/)
                h = {}
                matches.each { |x|
                    h[x[1]] ||= 0
                    h[x[1]] += x[0].to_i
                }
                event.set("buckets", h)
            end
        '
    }

will produce

    "buckets" => {
    "1" => 70,
    "0" => 30
},

[parinitha.nagaraja]

@Badger - Thank you, that was a good/simple solution.
A follow up question - How do we apply KV filter if the data is not in "key:value key:value" format?
If the format is "value:key value:key", is there a way to swap the data first to make it "key:value key:value" and then apply KV filter?

[Badger]

@parinitha.nagaraja

If the format is "value:key value:key", is there a way to swap the data first to make it "key:value key:value" and then apply KV filter?

I cannot think of a filter that would do that, except a ruby filter with custom code.