Logshash filter plugin ruby code to remove duplicates from a json array of json address objects

Elastic Stack Logstash
2

Your sample JSON objects are not valid. I tested this using

{
"id": "id1",
"addresses": [ {"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"} ],
"field1": "f1",  "field2": "f2", "field3": "f3",
"index2_addresses": [
{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919193"},
{"address": "a21","address_2": "a22","city": "c2","state": "s2","zip": "12346","phone": "9191919192"}
],
"field5": ["f51", "f52"], "field6": "f6"
}

The first problem is that .uniq does not modify an array, it returns a new array, which you discard. You should try

uniq_addresses = merged_addresses.uniq  { |addr| addr["xaddress"] + addr["address_2"] + addr["city"] + addr["state"] + addr["zip"] }
event.set("addresses", uniq_addresses)

with that change the code will de-dup the two address arrays, ignoring variations in phone numbers.

We can see from the stack trace that the exception you get is happening within the .uniq call.

undefined method `+' for nil:NilClass

will only happen if addr["address"] is nil. If subsequent references are nil (e.g.

 + addr["address_2"]

then you would get

no implicit conversion of nil into String

You need to add error handling for such malformed addresses.

1 Like