Continuous transformation is not update

Elastic Stack Elasticsearch
1

I created a pivot continuous transformation to automatically count the data in each location. Each document has an 'updated_at' field which is a timestamp in epoch second format when each document was last modified.

After I updated many documents, I found that the created transformation had not updated its checkpoint, and documents in the destination index remained the same.

I also found that there are some 'operations_behind' when I get statistics of the transformation.

"checkpointing": {
        "last": {
          "checkpoint": 1,
          "timestamp_millis": 1702437669083,
          "time_upper_bound_millis": 1702437609083
        },
        "operations_behind": 25868,
        "changes_last_detected_at": 1702437669081,
        "last_search_time": 1702438506413
      }

I'm not sure how continuous transformation updates itself. Maybe I missing something.

Hope anyone help

2

Hello,

I came across a similar case, your documents may be indexed a little late, and when Transform checks for 'new' documents since the last checkpoint, it finds nothing. Wich date field do you use in your Transform for checking new fields ?

Try to increase the time between two checkpoints (Frequeny) to find out if you have this problem.

3

Hi, GinkoLucas

I used field type 'date' in the format 'epoch_second' that showed the last modified timestamp. I just changed 'frequency' to 5m, 15m, and 30m and still nothing happened except operation_behind that keeps increasing.

Maybe this is because of wrong timestamp format?

4

When I read the documentation, I couldn't find any information about the date format. But it would be interesting to change it (or create a new field) and see if the Transform works.