# Create KV-pairs for field not in key:value format and use ruby filter on KV pairs

**URL:** <https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220>\
**Category:** Logstash\
**Created:** [July 8, 2021, 9:01pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220 "2021-07-08T21:01:49Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![parinitha.nagaraja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parinitha.nagaraja/32/134054_2.png) [@parinitha.nagaraja](https://discuss.elastic.co/u/parinitha.nagaraja)\
**Post date:** [July 8, 2021, 9:01pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220/1 "2021-07-08T21:01:49Z")

</div>

Hello,

I am trying to apply KV filter to a field that does not have "key:value key:value" format. The format is "value(key) value(key) ...". Below are the details.

- 2021-06-28 19:06:15.848 [0000-XYZ] [appLog] partition 100 in 4 buckets: [10(0) 20(1) 20(0) 50(1)]

- I pass the log line through a grok filer and create a filed called [bucket\_data].  
[bucket\_data]: 10(0) 20(1) 20(0) 50(1)

- Goal : To create fields as below :  
bucket.0 : 30 (which is the sum of all values in bucket 0 i.e 10+20=30)  
bucket.1 : 70 (which is the sum of all values in bucket 1 i.e 20+50=70)

- How I think we can achieve this is by passing the [bucket\_data] into a KV filter and then to a ruby filter to compute the sum.

Questions:

1. How do I use the KV filter in this case ? The [bucket\_data] format is value(key) value(key) .... What should be the field\_split, value\_split, trim\_key, trim\_value ?

2. How to I apply ruby filter after this ? Please see the code below.

3. Is there a better approach? Please let me know.

```auto
if [bucket_data] {		
			kv {
				source => "[bucket_data]"
				field_split => ""
				value_split => ""
				trim_key => "\s"
				trim_value => "\s"			
				target => "[bucket]"
			}

			if [bucket] {
				ruby { 
					code => " 
						kv = event.get('[bucket]')					 
						kv.to_hash.each { |key,values|	
							sum = 0
							values.each { |val| sum+=val };	
							event.set('[bucket][' + key + ']', sum.to_f);						
						}
					"
				}
			}								
		}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 8, 2021, 11:13pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220/2 "2021-07-08T23:13:02Z")

</div>

> [@parinitha.nagaraja](#):
>
> How do I use the KV filter in this case ?

I would not use a kv filter for this. There are two values that have a key "20", so sometimes the value is an array. Working around that will complicate things too much. I would just do it in ruby

```
    grok { match => { "message" => "\[(?<bucket_data>[^]]+)\]$" } }
    ruby {
        code => '
            data = event.get("bucket_data")
            if data
                matches = data.scan(/(\d+)\((\d)\)/)
                h = {}
                matches.each { |x|
                    h[x[1]] ||= 0
                    h[x[1]] += x[0].to_i
                }
                event.set("buckets", h)
            end
        '
    }

```

will produce

```
    "buckets" => {
    "1" => 70,
    "0" => 30
},

```

---

<div class="post-metadata">

**Author:** ![parinitha.nagaraja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/parinitha.nagaraja/32/134054_2.png) [@parinitha.nagaraja](https://discuss.elastic.co/u/parinitha.nagaraja)\
**Post date:** [July 9, 2021, 4:17pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220/3 "2021-07-09T16:17:49Z")

</div>

@Badger - Thank you, that was a good/simple solution.  
A follow up question - How do we apply KV filter if the data is not in "key:value key:value" format?  
If the format is "value:key value:key", is there a way to swap the data first to make it "key:value key:value" and then apply KV filter?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 9, 2021, 4:44pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220/4 "2021-07-09T16:44:39Z")

</div>

> [@parinitha.nagaraja](#):
>
> If the format is "value:key value:key", is there a way to swap the data first to make it "key:value key:value" and then apply KV filter?

I cannot think of a filter that would do that, except a ruby filter with custom code.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2021, 4:45pm UTC](https://discuss.elastic.co/t/create-kv-pairs-for-field-not-in-key-value-format-and-use-ruby-filter-on-kv-pairs/278220/5 "2021-08-06T16:45:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.