# Duplicate entries while using mutate **URL:** \ **Category:** Logstash\ **Created:** [August 29, 2019, 7:54pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413 "2019-08-29T19:54:41Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![humartinez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/humartinez/32/46395_2.png) [@humartinez](https://discuss.elastic.co/u/humartinez)\ **Post date:** [August 29, 2019, 7:54pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/1 "2019-08-29T19:54:41Z")
Hi there! Im having a issue (random I think) when using mutate and split. This is my logstash config ``` input { kafka { bootstrap_servers => "localhost:9092" topics => ["topic"] } } filter { mutate { split => ["message",","] add_field => { "field-a" => "%{[message][0]}"} } mutate { split => ["message",","] add_field => { "field-b" => "%{[message][1]}"} } mutate { split => ["message",","] add_field => { "field-c" => "%{[message][2]}"} } mutate { split => ["message",","] add_field => { "field-d" => "%{[message][3]}"} } } output { elasticsearch { hosts => ["localhost:9200"] index => "index" } } ``` An Im getting at index tiem the following entries duplicated ``` hits" : [ { "_index" : "index", "_type" : "_doc", "_id" : "wvnd3mwB1Aphc7auPmoX", "_score" : 1.0, "_source" : { "field-c" : [ "11", "11", "11" ], "field-b" : [ "00211", "00211", "00211" ], "@timestamp" : "2019-08-29T19:32:16.745Z", "@version" : "1", "message" : [ "sgit", "00211", "11", "pendiente" ], "field-a" : [ "sgit", "sgit", "sgit" ], "field-d" : [ "pendiente", "pendiente", "pendiente" ] } } ``` I don't know why it's triplicates the values for each field. Take note that the message list gives me one value per field I also changed my logstash config to this, with the same results ``` filter { mutate { add_field => { "field-a" => "%{[message][0]}"} } mutate { add_field => { "field-b" => "%{[message][1]}"} } mutate { add_field => { "field-c" => "%{[message][2]}"} } mutate { add_field => { "field-d" => "%{[message][3]}"} } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 29, 2019, 9:31pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/2 "2019-08-29T21:31:25Z")
What does the original [message] field look like? (Save a copy of it in another field.) I would have written that as ``` mutate { split => { "message" => "," } add_field => { "field-a" => "%{[message][0]}" "field-b" => "%{[message][1]}" "field-c" => "%{[message][2]}" "field-d" => "%{[message][3]}" } } ``` ---
**Author:** ![humartinez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/humartinez/32/46395_2.png) [@humartinez](https://discuss.elastic.co/u/humartinez)\ **Post date:** [August 30, 2019, 12:41pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/3 "2019-08-30T12:41:50Z")
Yes at first I wrote the filter as you suggested but I have the same results, as you can see in the previous output the original field message is preserved and have the following data ``` "message" : [ "sgit", "00211", "11", "pendiente" ] ``` Maybe I think I've found the problem, but I need your opinion. I was having 3 config files similar to the one that I posted. Each of these files takes a different topic from a kafka queue, then performs the same filter (the 3 files) and finally indexes the output to a different index. When I was working with just one config file the filters worked well (without duplicating the values of the filter) but when I add the others two files, the filters added more values per field as I have showed you. Right now Im using just one file with tags to redirect the inputs to the correct output. I this a recommended config or should I use 2 pipelines? and if so why? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [August 30, 2019, 1:02pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/4 "2019-08-30T13:02:23Z")
> [@humartinez](#): > > you can see in the previous output the original field message is preserved No, that is the message field after it has been split, but that no longer matters. If you have three configuration files then they are concatenated and events go through all three. So you effectively have ``` mutate { split => { "message" => "," } } mutate { add_field => { "field-a" => "%{[message][0]}" "field-b" => "%{[message][1]}" "field-c" => "%{[message][2]}" "field-d" => "%{[message][3]}" } } mutate { add_field => { "field-a" => "%{[message][0]}" "field-b" => "%{[message][1]}" "field-c" => "%{[message][2]}" "field-d" => "%{[message][3]}" } } mutate { add_field => { "field-a" => "%{[message][0]}" "field-b" => "%{[message][1]}" "field-c" => "%{[message][2]}" "field-d" => "%{[message][3]}" } } ``` which certainly will result in field-\* being arrays with three entries. If each configuration file is self-contained then yes, you should run it in its own pipeline. ---
**Author:** ![humartinez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/humartinez/32/46395_2.png) [@humartinez](https://discuss.elastic.co/u/humartinez)\ **Post date:** [August 30, 2019, 1:08pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/5 "2019-08-30T13:08:37Z")
Thanks you so much I didn't know that the config would be concatenated! 👍👍 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 27, 2019, 1:18pm UTC](https://discuss.elastic.co/t/duplicate-entries-while-using-mutate/197413/6 "2019-09-27T13:18:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.