Hi,
Have you considered using continuous transform instead of a batch transform?
From docs:
You can decide whether you want the transform to run once or continuously. A batch transform is a single operation that has a single checkpoint. Continuous transforms continually increment and process checkpoints as new source data is ingested.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.