I created a pivot continuous transformation to automatically count the data in each location. Each document has an 'updated_at' field which is a timestamp in epoch second format when each document was last modified.
After I updated many documents, I found that the created transformation had not updated its checkpoint, and documents in the destination index remained the same.
I also found that there are some 'operations_behind' when I get statistics of the transformation.
I came across a similar case, your documents may be indexed a little late, and when Transform checks for 'new' documents since the last checkpoint, it finds nothing. Wich date field do you use in your Transform for checking new fields ?
Try to increase the time between two checkpoints (Frequeny) to find out if you have this problem.
I used field type 'date' in the format 'epoch_second' that showed the last modified timestamp. I just changed 'frequency' to 5m, 15m, and 30m and still nothing happened except operation_behind that keeps increasing.
When I read the documentation, I couldn't find any information about the date format. But it would be interesting to change it (or create a new field) and see if the Transform works.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.