Logstash - Removing duplicity from array

[vasek (Václav Šulc)]

Hello,
I would like to share the experience with removing duplicity from array in Logstash.

I used this approach before:
I was checking whether value was already being present in array.

  if ( [host][name] and ([host][name] not in [host][address])) {
    mutate {
      add_field => { "[host][address]" => "%{[host][name]}" }
    }
  }

  if ( [agent][hostname] and ([agent][hostname] not in [host][address])) {
    mutate {
      add_field => { "[host][address]" => "%{[agent][hostname]}" }
    }
  }  

With ruby we can remove duplicity by uniq function very easily.
We can remove checks for testing presence of some value in array,

  if ( [host][name] ) {
    mutate {
      merge => { "[host][address]" => "[host][name]" }
    }
  }

  if ( [agent][hostname] ) {
    mutate {
      merge => { "[host][address]" => "[agent][hostname]" }
    }
 } 

ruby {
    code => "

       array = event.get('[host][address]')
       is_array = array.kind_of?(Array)

       if is_array
         event.set('[host][address]', event.get('[host][address]').uniq)
       end

     "
  }

I hope it will save you some time.