# Logshash filter plugin ruby code to remove duplicates from a json array of json address objects **URL:** \ **Category:** Logstash\ **Created:** [July 21, 2024, 1:47pm UTC](https://discuss.elastic.co/t/logshash-filter-plugin-ruby-code-to-remove-duplicates-from-a-json-array-of-json-address-objects/363501 "2024-07-21T13:47:01Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![vvavad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvavad/32/136129_2.png) [@vvavad](https://discuss.elastic.co/u/vvavad)\ **Post date:** [July 21, 2024, 1:47pm UTC](https://discuss.elastic.co/t/logshash-filter-plugin-ruby-code-to-remove-duplicates-from-a-json-array-of-json-address-objects/363501/1 "2024-07-21T13:47:01Z")
I am writing a logstash job/pipeline which takes input from Elasticsearch index(index 1), in filter plugin reads another elasticSearch index(index2). Both of these idices have json data with array of address objects. I want to merge the addresses from both sources, remove duplicates and save back to elasticSearch index1. However the duplicate removal logic, I am getting error in Ruby code block. My logstash config: ```auto input { elasticsearch { hosts => ["http://localhost:9200"] index => "index1" docinfo => true } } filter { # Read the document from carrier-json index elasticsearch { hosts => ["http://localhost:9200"] index => "index2" query => "_id:%{[uniq_id]}" fields => { "addresses" => "index2_addresses" } } # Merge addresses and remove duplicates if [index2_addresses] { ruby { code => ' existing_addresses = event.get("addresses") || [] new_addresses = event.get("index2_addresses") || [] merged_addresses = (existing_addresses + new_addresses) merged_addresses.uniq { |addr| addr["address"] + addr["address_2"] + addr["city"] + addr["state"] + addr["zip"] } event.set("addresses", merged_addresses) ' } } } output { stdout { codec => rubydebug } elasticsearch { hosts => ["http://localhost:9200"] index => "index1" document_id => "%{[@metadata][_id]}" action => "update" doc_as_upsert => true } } ``` json data in index1 ```auto { "id": "id1", "addresses": [ {"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"}, ], "field1": "f1", "field2": "f2", "field3": "f3" } ``` json data in index2 ```auto { "id": "a1", "addresses": [ {"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"}, {"address": "a21","address_2": "a22","city": "c2","state": "s2","zip": "12346","phone": "9191919192"}, ], "field5": [{"f51", "f52",}], "field6": "f6" } ``` exception: ```auto 2024-07-21T19:14:00,528][ERROR][logstash.filters.ruby][main][2b9cf7fde68fdb86bdd62c0c5fff77ac30a16c61c3c0af8b95a8a775dc5123f6] Ruby exception occurred: undefined method `+' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):6:in `block in filter_method'", "org/jruby/RubyArray.java:3464:in `uniq'", "(ruby filter code):6:in `block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in `block in start_workers'"]} ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 21, 2024, 2:58pm UTC](https://discuss.elastic.co/t/logshash-filter-plugin-ruby-code-to-remove-duplicates-from-a-json-array-of-json-address-objects/363501/2 "2024-07-21T14:58:25Z")
Your sample JSON objects are not valid. I tested this using ```auto { "id": "id1", "addresses": [{"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919191"}], "field1": "f1", "field2": "f2", "field3": "f3", "index2_addresses": [ {"address": "a11","address_2": "a21","city": "c1","state": "s1","zip": "12345","phone": "9191919193"}, {"address": "a21","address_2": "a22","city": "c2","state": "s2","zip": "12346","phone": "9191919192"} ], "field5": ["f51", "f52"], "field6": "f6" } ``` The first problem is that .uniq does not modify an array, it returns a new array, which you discard. You should try ``` uniq_addresses = merged_addresses.uniq { |addr| addr["xaddress"] + addr["address_2"] + addr["city"] + addr["state"] + addr["zip"] } event.set("addresses", uniq_addresses) ``` with that change the code will de-dup the two address arrays, ignoring variations in phone numbers. We can see from the stack trace that the exception you get is happening within the .uniq call. ``` undefined method `+' for nil:NilClass ``` will only happen if addr["address"] is nil. If subsequent references are nil (e.g. ``` + addr["address_2"] ``` then you would get ``` no implicit conversion of nil into String ``` You need to add error handling for such malformed addresses. ---
**Author:** ![vvavad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vvavad/32/136129_2.png) [@vvavad](https://discuss.elastic.co/u/vvavad)\ **Post date:** [July 22, 2024, 12:19pm UTC](https://discuss.elastic.co/t/logshash-filter-plugin-ruby-code-to-remove-duplicates-from-a-json-array-of-json-address-objects/363501/3 "2024-07-22T12:19:33Z")
@Badger thanks for clarification. It helped me root cause the issue.