# Logstash Mapping - Duplicate values in nested properties

**URL:** <https://discuss.elastic.co/t/logstash-mapping-duplicate-values-in-nested-properties/333554>\
**Category:** Logstash\
**Created:** [May 16, 2023, 10:05am UTC](https://discuss.elastic.co/t/logstash-mapping-duplicate-values-in-nested-properties/333554 "2023-05-16T10:05:58Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kgazula](https://avatars.discourse-cdn.com/v4/letter/k/258eb7/32.png) [@kgazula](https://discuss.elastic.co/u/kgazula)\
**Post date:** [May 16, 2023, 10:05am UTC](https://discuss.elastic.co/t/logstash-mapping-duplicate-values-in-nested-properties/333554/1 "2023-05-16T10:05:58Z")

</div>

Hello, can someone please help with mapping when there are more than 1 nested type properties in the mapping? We are using the 8.0 version and using Logstash we are synching the data from our Database to the ES index.

**Problem:** : I am seeing a duplicate of data getting created in the document for the nested type properties, when in the Logstash Config file I am mapping more than 1 nested type properties. Let me try to explain better with a sample example below.

**Index Mapping**

```auto
PUT test
{
  "settings": {
    "index.mapping.coerce": false
  },
  "mappings": {
    "dynamic": "strict",
    "properties" : {
	"agreementId" : {
          "type" : "text",
          "copy_to" : [
            "primaryFields"
          ]
        },
        "customers" : {
          "properties" : {
            "customerId" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
	    "customerAddresses" : {
              "type" : "nested",
              "properties" : {
                "custAddress" : {
                  "type" : "text"
                },
                "custAddressType" : {
                  "type" : "keyword",
                  "doc_values" : false
                }
              }
            },
	    "phones" : {
              "properties" : {
                "phonenumber" : {
                  "type" : "text",
                  "copy_to" : [
                    "primaryFields"
                  ]
                },
                "phonetype" : {
                  "type" : "keyword",
                  "doc_values" : false
                }
              }
            }
          }
        }
	}
  }
}

```

In my database, we have an Agreement number as the primary key that can have more than 1 customer profile (Let's use 1 in this scenario). Each customer can have multiple phones and multiple addresses. Based on the query, my output looks something like this

```auto
**agreement**  **customer**  **Address**  **Addresstype**  **Contact**  **Contacttype**
  123456879 10 123 Main St. Mailing 1111111111 Home
  123456789 10 123 Main St. Mailing 2222222222 Cell
  123456789 10 456 South Billing 1111111111 Home
  123456789 10 456 South Billing 2222222222 Cell

```

When the document is created in the Index, this is how it's looking

```auto
{
    "took": 474,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": null,
        "hits": [
            {
                "_index": "test",
                "_id": "123456789",
                "_score": null,
                "_source": {
                    "agreementId": 123456789,
                    "customers": [
                        {
                            "phones": [
                                {
                                    "phonetype": "Cell",
                                    "phonenumber": "2222222222"
                                },
                                {
                                    "phonetype": "Cell",
                                    "phonenumber": "2222222222"
                                },
                                {
                                    "phonetype": "Home",
                                    "phonenumber": "1111111111"
                                },
                                {
                                    "phonetype": "Home",
                                    "phonenumber": "1111111111"
                                }
                            ],
                            "customerAddresses": [
                                {
                                    "custAddressType": "Mailing",
                                    "custAddress": "123 Main St."
                                },
                                {
                                    "custAddressType": "Billing",
                                    "custAddress": "456 South"
                                },
                                {
                                    "custAddressType": "Mailing",
                                    "custAddress": "123 Main St."
                                },
                                {
                                    "custAddressType": "Billing",
                                    "custAddress": "456 South"
                                }
                            ]
                        }
                    ]
				},
                "sort": [
                    1713679200000
                ]
            }
        ]
    }
}

```

As you can see, the phones and customer addresses are getting repeated. Here is how the mapping is defined in the Config file.

**Config Mapping**

```auto
aggregate {
        task_id => "%{agreement}"
        code => "
                        map['agreementId'] = event.get('agreement')                       
                        
                         map['customers'] ||= []
                        if (event.get('customer') != nil)

                                customer_found = false
                                map['customers'].each { |cus|
                                        if cus['customerId'] == event.get('customer')
                                                customer_found = true
                                        end
                                }

                                if !customer_found
                                        map['customers'] << {
                                        'customerId' => event.get('customer')                          
                                        }
                                end
                                
                                map['customers'].each { |cus|
                                        if cus['customerId'] == event.get('customer') && event.get('Contact') != nil
                                                cus['phones'] ||=[]
                                                cus['phones'] << {
                                                'phonenumber' => event.get('Contact'),
                                                'phonetype' => event.get('Contacttype'),
                                                }
                                        end
                                }
                                
                                map['customers'].each { |cus|
                                        if cus['customerId'] == event.get('customer_id') && event.get('Address') != nil
                                                cus['customerAddresses'] ||=[]
                                                cus['customerAddresses'] << {
                                                'custAddress' => event.get('Address'),
                                                'custAddressType' => event.get('Addresstype'),
                                                }
                                        end
                                }
                        end
                                       
                        event.cancel()
            "
             push_previous_map_as_event => true
             timeout => 5
             timeout_tags => ['aggregated']
    }
    if "aggregated" not in [tags] {
            drop {}
        }
}

```

I even tried mentioning the "Phones" properties as Nested type, but no luck in the duplication.

@stephenb

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 13, 2023, 10:06am UTC](https://discuss.elastic.co/t/logstash-mapping-duplicate-values-in-nested-properties/333554/2 "2023-06-13T10:06:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.