# Kv Filter - allow\_duplicate\_values different behavior
**URL:** https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736
**Category:** Logstash
**Created:** [March 30, 2020, 7:17pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736 "2020-03-30T19:17:12Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)
#### Post date: [March 30, 2020, 7:17pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/1 "2020-03-30T19:17:12Z")
Hi guys,
Thanks for taking the time to read these lines,
I've got a question regarding `kv` filter and its `allow_duplicate_values` attribute
Assuming I have a log that looks like :
```
key1=value1|key2=value2|key3=value3|key1=value1|key2=value2|key3=value3
```
At the first place, it thought I always had keyX=valueX (most of the time the case)
So by using :
```auto
field_split => "|"
value_split => "="
allow_duplicate_values => false
```
I got the below :
```
{
key1 : "value1",
key2 : "value2",
key3 : "value3"
}
```
that was fine until I realized that sometimes I can have a `value4` in my 2nd `key1`
I'd like to understand why in this case my output looks like :
```
{
key1 : "value1,value4",
key2 : "value2",
key3 : "value3"
}
```
1/ How come ? I thought `allow_duplicate_values=false` would have kept either value1 or value4 but not both
2/ Actually, it would appear that I'm happy with this 2nd output, but is there a way to change the separator `,` and use `:` ?
Thank you
Guillaume
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 30, 2020, 8:56pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/2 "2020-03-30T20:56:10Z")
1. No, if you have
2. key1 should be an array. Are you converting it to string? If you want to change the separator in the string then use mutate+gsub.
---
### Author:  [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)
#### Post date: [March 30, 2020, 9:58pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/3 "2020-03-30T21:58:01Z")
Hi @Badger
Thanks for the reply,
1/ Sorry, what do you mean by "you always get both". In which format would I get both values ? `from: "Badger,Badger"` ?
2/ Downstream in the same logstash conf I'm doing :
```
add_field => { "a_new_key" => "%{key2}:%{key1}" }
```
So it was fine until I start seeing a few : `a_new_key: "value2:value1,value4"` (taking back my example of my post)
Do you think the conversion to string would have added the comma ?
I'm ideally looking for a way to get `a_new_key: "value2:value1"` when I have twice key1=value1 in my logs and `a_new_key: "value2:value1:value4"` when I have `key1=value1|...|key1=value4`
Thanks
Guillaume
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 30, 2020, 11:04pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/4 "2020-03-30T23:04:13Z")
When I say you always get both I mean if the two values are different, like
```
from=Badger from=GitsBdr
```
Yes, doing the add\_field with a sprintf reference to an array would do a to\_s on it, which would put join the members using comma as a separator.
```
mutate { gsub => ["message", ",", ":"] }
```
---
### Author:  [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)
#### Post date: [March 31, 2020, 8:57am UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/5 "2020-03-31T08:57:33Z")
Morning,
Lovely ! `gsub` does the trick perfectly thanks !
If I can bother you with one more question, how would Elasticsearch react to `from=Badger from=GitsBdr` ? It would first create the document with a field `from` and the value `Badger` and then update it with the value `GitsBdr` ?
And Logstash ? I quite don't get how would you separately manipulate these 2 values if they have the same key name !
thanks
Guillaume
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [March 31, 2020, 3:39pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/6 "2020-03-31T15:39:51Z")
In logstash kv produces an array by default if there are multiple occurrences of a key. If given an array elasticsearch will maintain it as an array.
---
### Author:  [@GitsBdr](https://discuss.elastic.co/u/GitsBdr)
#### Post date: [March 31, 2020, 3:49pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/7 "2020-03-31T15:49:47Z")
Alright makes sense. Thanks a lot 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 28, 2020, 3:49pm UTC](https://discuss.elastic.co/t/kv-filter-allow-duplicate-values-different-behavior/225736/8 "2020-04-28T15:49:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.