Exclude if not in array

[henrilabarre (Henri)]

Hi here is my source:

{
"telephone" => {
        "fine" => {
             "date" => "2017-10-26T16:54:28.477Z",
            "value" => "0387931080"
        }
    },
"TEL" => "0387931080",
}

I use this conf file to add the field telephone/fine/value and TEL to the same field:

filter {
  if "TEL" not in [TEL2] {
	  mutate {
		 merge => { "TEL2" => "TEL" }
	  }
  }
  if "telephone[fine][value]" not in [TEL2] {
	  mutate {
		 merge => { "TEL2" => "telephone[fine][value]" }
	  }
  }
  mutate {
    join => { "TEL2" => "," }
  }
}
output {
  stdout { codec => rubydebug }
}

The idea is to have a field with unique value of TEL, but I get duplicate:

"TEL2" => "0387931080,0387931080",

Thanks for point me what I miss!

[paz]

Your field reference is slightly malformed. Try this instead:

filter {
    if [TEL] not in [TEL2] {
	    mutate {
		    merge => { "TEL2" => "TEL" }
	    }
    }
    if [telephone][fine][value] not in [TEL2] {
	    mutate {
		    merge => { "TEL2" => "telephone[fine][value]" }
	    }
    }
    mutate {
        join => { "TEL2" => "," }
    }
}
output {
    stdout { codec => rubydebug }
}

You can also replicate that exact function with some ruby code, where you can use Sets (that by default only contain unique values), in order to avoid checks so the code is more compact.

filter {
    ruby {
        init => "require 'set'"
        code => "
            event.set('TEL2', Set.new([event.get('TEL'), event.get('[telephone][fine][value]')]).to_a.join(','))
        "
    }
}

[henrilabarre (Henri)]

Bravo and thanks for your help! It's working now