# Merge two datasets

**URL:** <https://discuss.elastic.co/t/merge-two-datasets/240506>\
**Category:** Elasticsearch\
**Created:** [July 9, 2020, 9:59am UTC](https://discuss.elastic.co/t/merge-two-datasets/240506 "2020-07-09T09:59:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![lugano](https://avatars.discourse-cdn.com/v4/letter/l/f9ae1b/32.png) [@lugano](https://discuss.elastic.co/u/lugano)\
**Post date:** [July 9, 2020, 9:59am UTC](https://discuss.elastic.co/t/merge-two-datasets/240506/1 "2020-07-09T09:59:53Z")

</div>

I have two datasets which I want to merge into one. I am using Elastic stack v7.7 . I want to use composite key with Painless script processor. To get a unique document I need 4 fields. 3 of the fields they match exactly for both datasets. The fourth fields match with only some of the documents but not all of them.  
eg. Transactions: cust\_ID, Name, Device, Date  
Preferences: cust\_ID, Name, Device, timestamp

The fields 'Date' and 'timestamp' they match only some of the time, therefore leaving out some of the data.

I used ingest pipeline of script processor to create the composite key as shown below:

PUT \_ingest/pipeline/preference\_pipe  
{  
"processors": [  
{  
"script": {  
"lang": "painless",  
"source": "ctx.composite\_key = ctx.cust\_ID + '#' + ctx.Name + '#' + ctx.Device + '#' + ctx.Date"  
}  
}  
]  
}

This created the composite key, but that key can not retrieve all the data as explained.

Is there any other way I can solve this ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 6, 2020, 10:04am UTC](https://discuss.elastic.co/t/merge-two-datasets/240506/2 "2020-08-06T10:04:31Z")

</div>

Closed on August 6, 2020, 10:04am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.