I have an issue with the transform i have created. It is aggregating on the source index which has events of the messages in it. The transform is running in continuous mode with the following options:
"sync": {
"time": {
"field": "modifiedAt",
"delay": "15s"
}
}
group by is as follows:
type or p"group_by": {
"campaignId": {
"terms": {
"field": "campaign.id"
}
},
"campaignTypeId": {
"terms": {
"field": "campaign.typeId"
}
},
"date": {
"date_histogram": {
"field": "createdAt",
"calendar_interval": "day"
}
}
},
the source index records are fine. Their modifiedAt date is updated accordingly as well, but
the transform is creating unexpected results. it is leaving out some document which should be processed using the groupBy field, that is only happening when the new document is added to index. The newest ones are usually aggregated, but the older ones get left out, i have run the same aggreggation i am doing in the transform and the result is as expected. but the transform is giving me a different one. i have done terms aggreagtion on the ids of the dpcument to see which one of them were being processed and i observed that some of the document were left out for sure. Also if i reset the transform then the transform gives the correct result.
What could be the problem?
Thanks in advance