I have two datasets which I want to merge into one. I am using Elastic stack v7.7 . I want to use composite key with Painless script processor. To get a unique document I need 4 fields. 3 of the fields they match exactly for both datasets. The fourth fields match with only some of the documents but not all of them.
eg. Transactions: cust_ID, Name, Device, Date
Preferences: cust_ID, Name, Device, timestamp
The fields 'Date' and 'timestamp' they match only some of the time, therefore leaving out some of the data.
I used ingest pipeline of script processor to create the composite key as shown below:
PUT _ingest/pipeline/preference_pipe
{
"processors": [
{
"script": {
"lang": "painless",
"source": "ctx.composite_key = ctx.cust_ID + '#' + ctx.Name + '#' + ctx.Device + '#' + ctx.Date"
}
}
]
}
This created the composite key, but that key can not retrieve all the data as explained.
Is there any other way I can solve this ?