# How to use the ruby{} filter to loop through k,v pairs after use the kv{} filter?

**URL:** <https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665>\
**Category:** Logstash\
**Created:** [September 7, 2018, 6:57am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665 "2018-09-07T06:57:15Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sun\_changlong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sun_changlong/32/39036_2.png) [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\
**Post date:** [September 7, 2018, 6:57am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/1 "2018-09-07T06:57:15Z")

</div>

After I use the kv plugin to parse the log, I want to filter the value further. What should I do?  
my data look like below:  
`level="2" treatment="3" cmd1="\"D:\Program Files (x86)\a\safe\modules\setup.exe\" /s /smartsilence" type="sys"`

the logstash config look like below:

```
filter {
  if[type] == "sys" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
      target => "kv"
    }  
  }

```

the result is 🙂

```auto
level="2"
treatment="3"
cmd="\"D:\Program Files (x86)\a\safe\modules\setup.exe\" /s /smartsilence"
type="sys"

```

I want to further process the value of cmd. Replace `\"` with `"`. so i add the ruby config look like below;

```
  ruby {
    code => "
      event = event.get['kv']
      event.to_hash.keys.each { |k, v|
       ...
      }
    "
  }
}

```

Is this method correct?  
how can I continue to process kv parsed data using ruby?

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [September 10, 2018, 1:07pm UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/2 "2018-09-10T13:07:16Z")

</div>

Do you only need to process a single field? If so, there is no need to iterate over all fields. Something like the below should suffice :

```auto
  ruby {
    code => "
      event.set('[kv][cmd]', event.get('[kv][cmd]').gsub('\"','"'))
    "
  }
}
```

---

<div class="post-metadata">

**Author:** ![sun\_changlong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sun_changlong/32/39036_2.png) [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\
**Post date:** [September 11, 2018, 3:31am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/3 "2018-09-11T03:31:09Z")

</div>

No, I need to filter all values。I used the target in the kv plugin, using ruby to pick up the target for parsing and filtering, but the target changed the original field name.

the setting loog like below:

```
filter {
  if[type] == "syslog" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
      target => "kv"
    }
  }

  ruby {
    code => "
      event = event.get('kv')
      event.each { |key,value|
        if value.include? '\"' 
          event[key] = value.gsub!('\"', '')
        end
      }
   "
}

```

The original field name might be **detail.name** , but it now becomes **kv.detail.name**.  
What should I do if I don't use **target** as a connection? Or after using the **targe** t in ruby，how to extract the data in the target to the root directory?

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [September 11, 2018, 9:17am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/4 "2018-09-11T09:17:40Z")

</div>

Oh, I see.  
If you don't use define a _target_ field for the KV filter, any created field is automatically created in the event root level, which is probably what you want.

You can then iterate over all keys like so :

```auto
filter {
  if [type] == "syslog" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
    }
  }

  ruby {
    code => "
      hash = event.to_hash
      hash.each { |key,value|
        if value.include? '\"' 
          event.set(key, value.gsub!('\"', ''))
        end
      }
   "
}
```

---

<div class="post-metadata">

**Author:** ![sun\_changlong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sun_changlong/32/39036_2.png) [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\
**Post date:** [September 12, 2018, 2:34am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/5 "2018-09-12T02:34:09Z")

</div>

Thank you very much, it can solve my problem。But I have a question, for the timestamp field, ruby will report parsing exceptions, I used the if method of filtering, is there a better way?

> [ERROR][logstash.filters.ruby] Ruby exception occurred: undefined method `include?' for 2018-09-12T02:28:45.968Z:LogStash::Timestamp

the field is like` "@timestamp" => 2018-09-12T02:28:45.969Z`

I setting the config look like below:

```
ruby {
   code => "
      hash = event.to_hash
      hash.each { |key,value|
        if key != '@timestamp' and value.include? '\"'
          event.set(key, value.gsub!('\"', ''))
        end
      }
   "
}

```

Filter through by `if key != '@timestamp' and value.include? '\"'`

---

<div class="post-metadata">

**Author:** ![paz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paz/32/28003_2.png) [@paz](https://discuss.elastic.co/u/paz)\
**Post date:** [September 13, 2018, 9:24am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/6 "2018-09-13T09:24:07Z")

</div>

You can exclude @timestamp this way, sure. You can also try this to only apply it to strings, but not sure it will work as intended.

```auto
if value.is_a?(String) && value.include?('\"')
```

---

<div class="post-metadata">

**Author:** ![sun\_changlong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sun_changlong/32/39036_2.png) [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)\
**Post date:** [September 14, 2018, 1:31am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/7 "2018-09-14T01:31:14Z")

</div>

Thanks again,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 12, 2018, 1:31am UTC](https://discuss.elastic.co/t/how-to-use-the-ruby-filter-to-loop-through-k-v-pairs-after-use-the-kv-filter/147665/8 "2018-10-12T01:31:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.