We have intraday updates to our records, so we need to delete and insert all child records.
logstash output : This is not deleting and insert , it's insert or update.
output {
elasticsearch{
hosts => [ "xxxxxx" ]
index => "xxxxxx"
document_id => "xxxxxx"
document_type => "_doc"
user => xxxxx
password => xxxxx
action => "update"
doc_as_upsert => "true"
}
}
sample:
initial data in elastic:
"keyInformation" : {
"firmId" : "1",
"entityType" : "HH",
"entityId" : "1111",
"lastUpdated" : "2020-04-09T16:08:17.181Z",
"key" : "QA8_COMMON_101_HH_111"
},
"associatedAccounts" : [
{
"accountId" : "1",
},
{
"accountId" : "2",
},
{
"accountId" : "3",
}
]
after intraday load it should remove 1,2,3 and add 3,4,5
"associatedAccounts" : [
{
"accountId" : "4",
},
{
"accountId" : "5",
},
{
"accountId" : "6",
}
we can't remove entire document because there are other child objects that may not have changed. so we have to update only specific list.