I need to create a transform that will process the data from another transform's destination index to further aggregate the aggregated data. It works fine as a one-time job, but can it work continuously?
Considering that the original transform uses date_histogram
to aggregate data by days, is it possible to insert a "current timestamp" field into the original destination index documents and use it for "sync" in the second-level transform? I'm assuming it should be possible using an Ingest pipeline. But when an index is updated by a transform, is it really updated or recreated (deleted and inserted)? So, will the "current timestamp" also be updated and trigger the second transform to process the update?
Or are there any other recommendations regarding how further aggregating the transformed data can be accomplished? Thanks!