How to use the ruby{} filter to loop through k,v pairs after use the kv{} filter?

[sun_changlong]

After I use the kv plugin to parse the log, I want to filter the value further. What should I do?
my data look like below:
level="2" treatment="3" cmd1="\"D:\Program Files (x86)\a\safe\modules\setup.exe\" /s /smartsilence" type="sys"

the logstash config look like below:

filter {
  if[type] == "sys" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
      target => "kv"
    }  
  }

the result is

level="2"
treatment="3"
cmd="\"D:\Program Files (x86)\a\safe\modules\setup.exe\" /s /smartsilence"
type="sys"

I want to further process the value of cmd. Replace \" with ". so i add the ruby config look like below;

  ruby {
    code => "
      event = event.get['kv']
      event.to_hash.keys.each { |k, v|
       ...
      }
    "
  }
}

Is this method correct?
how can I continue to process kv parsed data using ruby?

[paz]

Do you only need to process a single field? If so, there is no need to iterate over all fields. Something like the below should suffice :

  ruby {
    code => "
      event.set('[kv][cmd]', event.get('[kv][cmd]').gsub('\"','"'))
    "
  }
}

[sun_changlong]

No, I need to filter all values。I used the target in the kv plugin, using ruby to pick up the target for parsing and filtering, but the target changed the original field name.

the setting loog like below:

filter {
  if[type] == "syslog" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
      target => "kv"
    }
  }

  ruby {
    code => "
      event = event.get('kv')
      event.each { |key,value|
        if value.include? '\"' 
          event[key] = value.gsub!('\"', '')
        end
      }
   "
}

The original field name might be detail.name, but it now becomes kv.detail.name.
What should I do if I don't use target as a connection? Or after using the target in ruby，how to extract the data in the target to the root directory?

[paz]

Oh, I see.
If you don't use define a target field for the KV filter, any created field is automatically created in the event root level, which is probably what you want.

You can then iterate over all keys like so :

filter {
  if [type] == "syslog" {
    kv {
      source => "message"
      field_split => "[,\s]"
      value_split => "="
    }
  }

  ruby {
    code => "
      hash = event.to_hash
      hash.each { |key,value|
        if value.include? '\"' 
          event.set(key, value.gsub!('\"', ''))
        end
      }
   "
}

[sun_changlong]

Thank you very much, it can solve my problem。But I have a question, for the timestamp field, ruby will report parsing exceptions, I used the if method of filtering, is there a better way?

[ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `include?' for 2018-09-12T02:28:45.968Z:LogStash::Timestamp

the field is like "@timestamp" => 2018-09-12T02:28:45.969Z

I setting the config look like below:

ruby {
   code => "
      hash = event.to_hash
      hash.each { |key,value|
        if key != '@timestamp' and value.include? '\"'
          event.set(key, value.gsub!('\"', ''))
        end
      }
   "
}

Filter through by if key != '@timestamp' and value.include? '\"'

[paz]

You can exclude @timestamp this way, sure. You can also try this to only apply it to strings, but not sure it will work as intended.

if value.is_a?(String) && value.include?('\"')

[sun_changlong]

Thanks again,