How to remove duplicate values in logstash

[Sajal (Sajal Sachan)]

Hi .

Unable to remove duplicate values from logstash .
"src" => [
[0] "10.207.73.1",
[1] "10.207.73.1"
],
"agentName" => [
[0] "sl73tdsdb1q01 ",
[1] "sl73tdsdb1q01 "
],
"StoredProc" => [
[0] "False",
[1] "False"
],
"LEEF:1.0|Imperva|SecureSphere|12.4.0.80_0|DeviceType" => "ImpervaDAMaudit|ImpervaVersion=12.4.0.80_0|Policy=VISA",
"srvGroup" => [
[0] "VBV",
[1] "VBV"
],
"2.0|ApplicationName" => "Default",
"beat" => {
"version" => "6.6.1",
"name" => "sl73splnkapp085.visa.com",
"hostname" => "sl73splnkapp085.visa.com"
},
"Objecttype" => [
[0] "{Event.struct.operations.objectType}", [1] "{Event.struct.operations.objectType}"
],
"SrvName" => [
[0] "Oracle Database 3DS 2.0",
[1] "Oracle Database 3DS 2.0"
],

[Jenni]

I think that should to the trick:

ruby {
  code => "
    event.set('src', event.get('src').uniq)
    event.set('agentName', event.get('agentName').uniq)
    [.. etc. …]
  "
}

For all fields of the event it should be something like this, I guess:

event.to_hash.each { |k, v | event.set(k, v.uniq) if v.is_a?Array }