Create KV-pairs for field not in key:value format and use ruby filter on KV pairs

Elastic Stack Logstash
1

Hello,

I am trying to apply KV filter to a field that does not have "key:value key:value" format. The format is "value(key) value(key) ...". Below are the details.

  • 2021-06-28 19:06:15.848 [0000-XYZ] [appLog ] partition 100 in 4 buckets: [ 10(0) 20(1) 20(0) 50(1) ]

  • I pass the log line through a grok filer and create a filed called [bucket_data].
    [bucket_data]: 10(0) 20(1) 20(0) 50(1)

  • Goal : To create fields as below :
    bucket.0 : 30 (which is the sum of all values in bucket 0 i.e 10+20=30)
    bucket.1 : 70 (which is the sum of all values in bucket 1 i.e 20+50=70)

  • How I think we can achieve this is by passing the [bucket_data] into a KV filter and then to a ruby filter to compute the sum.

Questions:

  1. How do I use the KV filter in this case ? The [bucket_data] format is value(key) value(key) .... What should be the field_split, value_split, trim_key, trim_value ?

  2. How to I apply ruby filter after this ? Please see the code below.

  3. Is there a better approach? Please let me know.

if [bucket_data] {		
			kv {
				source => "[bucket_data]"
				field_split => ""
				value_split => ""
				trim_key => "\s"
				trim_value => "\s"			
				target => "[bucket]"
			}

			if [bucket] {
				ruby { 
					code => " 
						kv = event.get('[bucket]')					 
						kv.to_hash.each  { |key,values|	
							sum = 0
							values.each { |val| sum+=val };	
							event.set('[bucket][' + key + ']', sum.to_f);						
						}
					"
				}
			}								
		}
2

I would not use a kv filter for this. There are two values that have a key "20", so sometimes the value is an array. Working around that will complicate things too much. I would just do it in ruby

    grok { match => { "message" => "\[(?<bucket_data>[^]]+)\]$" } }
    ruby {
        code => '
            data = event.get("bucket_data")
            if data
                matches = data.scan(/(\d+)\((\d)\)/)
                h = {}
                matches.each { |x|
                    h[x[1]] ||= 0
                    h[x[1]] += x[0].to_i
                }
                event.set("buckets", h)
            end
        '
    }

will produce

    "buckets" => {
    "1" => 70,
    "0" => 30
},
1 Like
3

@Badger - Thank you, that was a good/simple solution.
A follow up question - How do we apply KV filter if the data is not in "key:value key:value" format?
If the format is "value:key value:key", is there a way to swap the data first to make it "key:value key:value" and then apply KV filter?

4

I cannot think of a filter that would do that, except a ruby filter with custom code.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.