Merge two datasets

I have two datasets which I want to merge into one. I am using Elastic stack v7.7 . I want to use composite key with Painless script processor. To get a unique document I need 4 fields. 3 of the fields they match exactly for both datasets. The fourth fields match with only some of the documents but not all of them.
eg. Transactions: cust_ID, Name, Device, Date
Preferences: cust_ID, Name, Device, timestamp

The fields 'Date' and 'timestamp' they match only some of the time, therefore leaving out some of the data.

I used ingest pipeline of script processor to create the composite key as shown below:

PUT _ingest/pipeline/preference_pipe
"processors": [
"script": {
"lang": "painless",
"source": "ctx.composite_key = ctx.cust_ID + '#' + ctx.Name + '#' + ctx.Device + '#' + ctx.Date"

This created the composite key, but that key can not retrieve all the data as explained.

Is there any other way I can solve this ?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.