Transform Retention Policy

Elastic Stack Elasticsearch
1

Hello,

I was hoping for some clarity around transforms' retention policy. My understanding is that any documents in the destination index would be "ignored" from the pivot computations, when the retention policy datetime field's age exceeds the threshold. However, I'm experiencing those expired documents still being included in the transformations.

Transform Retention Policy Configuration:

"retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "1h"
    }
  }

Query for count of documents within the last hour: (Results in no hits)

{
    "query": {
        "range": {
            "order_date": {
                "gte": "now-1h/h"
            }
        }
    }
}

The transform is configured to check for changes in the source index using the ingest timestamp, approximately every 5-minutes.

"frequency": "5m",
"sync": {
  "time": {
    "field": "@timestamp",
    "delay": "60s"
  }
}

Since creating the transform, I have pushed documents to the source index -- in attempt to notify the transform of updates -- and waited a long while. The transformation continues to aggregate new metrics, without removing the old. All 8 documents shown below have an order_date greater than 1-hour.

Screen Shot 2022-01-06 at 10.54.34 AM
Screen Shot 2022-01-06 at 10.54.34 AM1822×366 27 KB

I'd appreciate any insight into what I may be doing wrong, or if I'm misinterpreting the usage of the retention policy.

Thank you in advance for any help!

2

I would like to add that including the datetime range query detailed above as a "bool-should minimum match" query in the transform does solve the expired document issue, but would prefer to use to the retention policy if possible, as it's easily configurable in Kibana -- even after creating the transformation.

3

Did your transform create new checkpoints? The retention policy is executed after every checkpoint, if no checkpoint is produced it is not executed. This is by design, e.g. max_age is not calculated from the current system time, but based on the time of the checkpoint.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.