Understanding operations_behind in transform stats

Elastic Stack Elasticsearch
1

Hello,

I have a question about the metric operations_behind when I call the transform stats API.

The elastic doc says the following about operations_behind :

The number of operations that have occurred on the source index but have not been applied to the destination index yet. A high number can indicate that the transform is failing to keep up.

Ok but what is an operation here ? Is it a "write" or "delete" for a document ?

I am using transform job of type "Latest". So if a log with the same key is ingested, the transform job will do an update on my output index. This update will be an insert of the new document and a delete of the old one. So what is the impact for operations_behind here ?

Regards, Eric

2

Ok but what is an operation here ? Is it a "write" or "delete" for a document ?

An operation is any call to add, update, or delete a document within the source index.

I am using transform job of type "Latest". So if a log with the same key is ingested, the transform job will do an update on my output index. This update will be an insert of the new document and a delete of the old one. So what is the impact for operations_behind here ?

Before the transform runs the next checkpoint, the ingested log will count as 1 operation_behind (the update of the log with the same key).

When the transform runs the next checkpoint, it will search for the changes and add a new document to the destination index with the results.

Operations should have a timestamp that occurs after the last checkpoint timestamp, otherwise changes may be missed, but the operations_behind will still be reset to 0: Transform limitations | Elasticsearch Guide [8.14] | Elastic

3

Thank you for your reply Patrick.

1 Like