# Transform each array item (nested object) in a new document - Splitting a field before removing it **URL:** \ **Category:** Logstash\ **Created:** [February 25, 2019, 2:34pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851 "2019-02-25T14:34:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Lucas\_Porto](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lucas\_Porto](https://discuss.elastic.co/u/Lucas_Porto)\ **Post date:** [February 25, 2019, 2:34pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/1 "2019-02-25T14:34:54Z")
Hi all, I have a index with a nested field like this: ``` { "id_single_profile": "14e98c40-d5ed-11e8-aa2a-d991a7f7e009", "id_audience": "3c9b46e4-1b10-4ad4-9a68-ae371034adfe", "activities": [ { "id_activity": "activity-1", "nm_product": "xx", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" }, { "id_activity": "activity-2", "nm_product": "yy", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" } ] } ``` So I'm trying to transform each nested object in a new index document. Something like this: ``` [ { "_index": "doc1", "_type": "activity", "_id": "activity-1", "_source": { "nm_product": "xx", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" } } { "_index": "doc1", "_type": "activity", "_id": "activity-2", "_source": { "nm_product": "yy", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" } } ] ``` -- I've been using this filter below in Logstash. I'm splitting by the "activities" nested field and this doing fine. The problem is to remove the nested field from the document after doing the split. When I use the "remove\_field" to remove the "activities" field from output the split just don't work. ``` filter { split { field => "activities" } mutate { add_field => { "nm_product" => "%{[activities][nm_product]}" "nm_product_category" => "%{[activities][nm_product_category]}" "nm_establishment_category" => "%{[activities][nm_establishment_category]}" } } mutate { remove_field => ["@version","@timestamp"] } } ``` Is there something I'm missing? I will appreciate any comments. Thanks! ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 25, 2019, 2:46pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/2 "2019-02-25T14:46:27Z")
Provided you do the remove\_field _after_ the split it works just fine for me. ---
**Author:** ![Lucas\_Porto](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lucas\_Porto](https://discuss.elastic.co/u/Lucas_Porto)\ **Post date:** [February 25, 2019, 2:52pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/3 "2019-02-25T14:52:44Z")
Didn't work for me. Adding a remove\_field to "activities" the split don't work and returns just one document. ``` filter { split { field => "activities" } mutate { remove_field => ["@version","@timestamp","activities"] } mutate { add_field => { "nm_product_category" => "%{[activities][nm_product_category]}" "nm_establishment_category" => "%{[activities][nm_establishment_category]}" "vl_total" => "%{[activities][vl_total]}" } } ``` } ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 25, 2019, 2:54pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/4 "2019-02-25T14:54:56Z")
Do the remove of activities after you have copied the fields out of it using mutate+add\_field. ---
**Author:** ![Lucas\_Porto](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lucas\_Porto](https://discuss.elastic.co/u/Lucas_Porto)\ **Post date:** [February 25, 2019, 3:00pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/5 "2019-02-25T15:00:10Z")
Yes, I have tried putting the remove before and after the copy of fields. Both didn't work. Same results. Seems like the order of the mutate's don't make any difference. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 25, 2019, 3:15pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/6 "2019-02-25T15:15:15Z")
> [@Lucas\_Porto](#): > > Seems like the order of the mutate's don't make any difference. Order most certainly matters. ---
**Author:** ![Lucas\_Porto](https://avatars.discourse-cdn.com/v4/letter/l/f05b48/32.png) [@Lucas\_Porto](https://discuss.elastic.co/u/Lucas_Porto)\ **Post date:** [February 25, 2019, 4:09pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/7 "2019-02-25T16:09:25Z")
I've tried using metadata fields for copy the values but I got the same results. I can split the nested object and add new fields based on the splitted fields. The problem is when I try to remove the field that I'm using to splitting. Seems like I can not split a field before removing it in the same pipeline. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 25, 2019, 4:43pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/8 "2019-02-25T16:43:54Z")
I cannot imagine why that would be. When I run ``` input { generator { count => 1 message => '{ "id_single_profile": "14e98c40-d5ed-11e8-aa2a-d991a7f7e009", "id_audience": "3c9b46e4-1b10-4ad4-9a68-ae371034adfe", "activities": [ { "id_activity": "activity-1", "nm_product": "xx", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" }, { "id_activity": "activity-2", "nm_product": "yy", "nm_product_category": "WEB", "nm_establishment_category": "Softwares / APP" } ] }' } } filter { json { source => "message" } split { field => "activities" } mutate { add_field => { "nm_product" => "%{[activities][nm_product]}" "nm_product_category" => "%{[activities][nm_product_category]}" "nm_establishment_category" => "%{[activities][nm_establishment_category]}" } } mutate { remove_field => ["@version", "@timestamp", "activities", "message"] } } output { stdout { codec => rubydebug { metadata => false } } } ``` I get two events ``` { "id_audience" => "3c9b46e4-1b10-4ad4-9a68-ae371034adfe", "nm_product" => "xx", "nm_establishment_category" => "Softwares / APP", "id_single_profile" => "14e98c40-d5ed-11e8-aa2a-d991a7f7e009", "nm_product_category" => "WEB", [...] } { "id_audience" => "3c9b46e4-1b10-4ad4-9a68-ae371034adfe", "nm_product" => "yy", "nm_establishment_category" => "Softwares / APP", "id_single_profile" => "14e98c40-d5ed-11e8-aa2a-d991a7f7e009", "nm_product_category" => "WEB", [...] } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 25, 2019, 4:43pm UTC](https://discuss.elastic.co/t/transform-each-array-item-nested-object-in-a-new-document-splitting-a-field-before-removing-it/169851/9 "2019-03-25T16:43:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.