Transform Retention Policy

[KentHHarris (Kent Harris)]

Hello,

I was hoping for some clarity around transforms' retention policy. My understanding is that any documents in the destination index would be "ignored" from the pivot computations, when the retention policy datetime field's age exceeds the threshold. However, I'm experiencing those expired documents still being included in the transformations.

Transform Retention Policy Configuration:

"retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "1h"
    }
  }

Query for count of documents within the last hour: (Results in no hits)

{
    "query": {
        "range": {
            "order_date": {
                "gte": "now-1h/h"
            }
        }
    }
}

The transform is configured to check for changes in the source index using the ingest timestamp, approximately every 5-minutes.

"frequency": "5m",
"sync": {
  "time": {
    "field": "@timestamp",
    "delay": "60s"
  }
}

Since creating the transform, I have pushed documents to the source index -- in attempt to notify the transform of updates -- and waited a long while. The transformation continues to aggregate new metrics, without removing the old. All 8 documents shown below have an order_date greater than 1-hour.

I'd appreciate any insight into what I may be doing wrong, or if I'm misinterpreting the usage of the retention policy.

Thank you in advance for any help!

[KentHHarris (Kent Harris)]

I would like to add that including the datetime range query detailed above as a "bool-should minimum match" query in the transform does solve the expired document issue, but would prefer to use to the retention policy if possible, as it's easily configurable in Kibana -- even after creating the transformation.

[Hendrik_Muhs]

@KentHHarris

Since creating the transform, I have pushed documents to the source index -- in attempt to notify the transform of updates -- and waited a long while.

Did your transform create new checkpoints? The retention policy is executed after every checkpoint, if no checkpoint is produced it is not executed. This is by design, e.g. max_age is not calculated from the current system time, but based on the time of the checkpoint.