# Transform Retention Policy

**URL:** <https://discuss.elastic.co/t/transform-retention-policy/293683>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [January 6, 2022, 7:09pm UTC](https://discuss.elastic.co/t/transform-retention-policy/293683 "2022-01-06T19:09:40Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![KentHHarris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenthharris/32/99929_2.png) [@KentHHarris](https://discuss.elastic.co/u/KentHHarris)\
**Post date:** [January 6, 2022, 7:09pm UTC](https://discuss.elastic.co/t/transform-retention-policy/293683/1 "2022-01-06T19:09:40Z")

</div>

Hello,

I was hoping for some clarity around transforms' retention policy. My understanding is that any documents in the destination index would be "ignored" from the pivot computations, when the retention policy datetime field's age exceeds the threshold. However, I'm experiencing those expired documents still being included in the transformations.

**Transform Retention Policy Configuration:**

```auto
"retention_policy": {
    "time": {
      "field": "order_date",
      "max_age": "1h"
    }
  }

```

**Query for count of documents within the last hour:** (Results in no hits)

```auto
{
    "query": {
        "range": {
            "order_date": {
                "gte": "now-1h/h"
            }
        }
    }
}

```

The transform is configured to check for changes in the source index using the ingest timestamp, approximately every 5-minutes.

```auto
"frequency": "5m",
"sync": {
  "time": {
    "field": "@timestamp",
    "delay": "60s"
  }
}

```

Since creating the transform, I have pushed documents to the source index -- in attempt to notify the transform of updates -- and waited a long while. The transformation continues to aggregate new metrics, without removing the old. All 8 documents shown below have an `order_date` greater than 1-hour.

 ![Screen Shot 2022-01-06 at 10.54.34 AM](https://us1.discourse-cdn.com/elastic/original/3X/5/5/55cbae0c05c396495512b839326ade292619bd12.png)

I'd appreciate any insight into what I may be doing wrong, or if I'm misinterpreting the usage of the retention policy.

Thank you in advance for any help!

---

<div class="post-metadata">

**Author:** ![KentHHarris](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kenthharris/32/99929_2.png) [@KentHHarris](https://discuss.elastic.co/u/KentHHarris)\
**Post date:** [January 6, 2022, 7:16pm UTC](https://discuss.elastic.co/t/transform-retention-policy/293683/2 "2022-01-06T19:16:46Z")

</div>

I would like to add that including the **datetime range query** detailed above _as_ a "bool-should minimum match" query in the transform does solve the expired document issue, but would prefer to use to the retention policy if possible, as it's easily configurable in Kibana -- even after creating the transformation.

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [January 10, 2022, 7:49am UTC](https://discuss.elastic.co/t/transform-retention-policy/293683/3 "2022-01-10T07:49:57Z")

</div>

> [@KentHHarris](#):
>
> Since creating the transform, I have pushed documents to the source index -- in attempt to notify the transform of updates -- and waited a long while.

Did your transform create new checkpoints? The retention policy is executed after every checkpoint, if no checkpoint is produced it is not executed. This is by design, e.g. `max_age` is not calculated from the current system time, but based on the time of the checkpoint.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 7, 2022, 7:50am UTC](https://discuss.elastic.co/t/transform-retention-policy/293683/4 "2022-02-07T07:50:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.