# Remove duplicates in a logstash array

**URL:** <https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705>\
**Category:** Logstash\
**Created:** [April 26, 2018, 3:02pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705 "2018-04-26T15:02:43Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![SKing](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SKing](https://discuss.elastic.co/u/SKing)\
**Post date:** [April 26, 2018, 3:02pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705/1 "2018-04-26T15:02:44Z")

</div>

I have a json String with name / value pairs

{  
"A1":"123",  
"A2":"225",  
"A3":"668",  
"A4":"225"  
}

Using logstash I can mutate the string and end up with an array by using  
mutate {  
add\_field =\> {A =\> "%{A1}")  
add\_field =\> {A =\> "%{A2}")  
add\_field =\> {A =\> "%{A3}")  
add\_field =\> {A =\> "%{A4}")  
}

mutate {  
remove\_field =\> ["A1", "A2", "A3", "A4"]  
}

{  
"A": ["123", "225", "668", "225"]  
}

I would like to eliminate the duplicates in the array so that I end up with  
{  
"A": ["123", "225", "668"]  
}

I have tried using kv {} with allow\_duplicate\_values=false  
but that would only work if my input data was of the format  
{  
"A":"123",  
"A":"225",  
"A":"668",  
"A":"225"  
}

Looking at this post

> [@Checking if an nested field which is an array contains a value](https://discuss.elastic.co/t/checking-if-an-nested-field-which-is-an-array-contains-a-value/55561):
>
> Hi: I am trying to check if a nested field in the input can be used for conditional checks and how to do it: here is an example logstash config : input { generator { message =\> '{"metadata": { "origins": ["192.168.50.91"] }}' codec =\> "json" count =\> 1 } } filter { if [metadata][origins] =~ /192\.168\.50\.91/ { mutate { add\_field =\> ["router.cisco.hostname", "foo"] } } } output { stdout { codec =\> rubydebug } } In the above config, I want to be abl…

it would appear to be some variation on  
if [A2] in [A] {  
if %{A2} in [A] {

Would appreciate people's help with this.  
Thanks

---

<div class="post-metadata">

**Author:** ![Jenni](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jenni/32/29684_2.png) [@Jenni](https://discuss.elastic.co/u/Jenni)\
**Post date:** [April 26, 2018, 3:25pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705/2 "2018-04-26T15:25:17Z")

</div>

```
ruby {
  code => 'event.set("A", event.get("A").uniq)'
}
```

---

<div class="post-metadata">

**Author:** ![SKing](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@SKing](https://discuss.elastic.co/u/SKing)\
**Post date:** [April 26, 2018, 3:32pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705/3 "2018-04-26T15:32:15Z")

</div>

Thanks Jenni.

Even quicker using the built in plug-ins

if [A1] {  
mutate {  
add\_field =\> {"A" =\> "%{A1}"}  
}  
}  
if [A2] {   
if !([A]) or !([A2] in [A]) {  
mutate {  
add\_field =\> {"A" =\> "%{A2}"}   
}   
}  
}  
if [A3] {   
if !([A]) or !([A3] in [A]) {  
mutate {  
add\_field =\> {"A" =\> "%{A3}"}   
}   
}  
}  
if [A4] {   
if !([A]) or !([A4] in [A]) {  
mutate {  
add\_field =\> {"A" =\> "%{A4}"}   
}   
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 24, 2018, 3:32pm UTC](https://discuss.elastic.co/t/remove-duplicates-in-a-logstash-array/129705/4 "2018-05-24T15:32:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.