# Elastic Transforms - continous mode is not detecting changes
**URL:** https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895
**Category:** Elasticsearch
**Tags:** transforms
**Created:** [January 25, 2023, 8:25am UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895 "2023-01-25T08:25:05Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@catrexis](https://discuss.elastic.co/u/catrexis)
#### Post date: [January 25, 2023, 8:25am UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895/1 "2023-01-25T08:25:05Z")
I have a problem with my index transformation and already tried so many different versions but nothing helped:
My documents in the source index look like:
```auto
{
"content" : {
"creationTime" : "2022-07-25 16:00:49 +02:00",
"orderId" : "552313",
"state" : "SUCCESS",
"category" : "catA"
},
"latestUpdate" : "2022-11-29 18:42:08 +01:00"
}
}
```
I want to use transforms to put the orders into time-buckets of 1minute (I originally wanted to use 30minutes, but I noticed that only 1 minute or 1 hour is supported but nothing in between, right?) and filter out orders with the state ERROR.
Every time, the data gets updated, the timestamp "latestUpdate" also changes to the current time.
My source index is updated every 10 minutes and overwrites some documents of the last week, since the status-field-entry changes.
My transform request looks like this:
```auto
PUT _transform/transform_my_index001
{
"source" : {
"index" : "my_index",
"query": {
"bool": {
"must_not":[
{"term":{"content.state":"ERROR"}}
]
}
}
}
},
"pivot": {
"group_by": {
"creation_time": {
"date_histogram": {
"field": "content.creationTime",
"calendar_interval": "1m",
"time_zone": "Europe/Berlin"
}
},
"portal": {
"terms": {
"field": "content.portal"
}
}
"category": {
"terms": {
"field": "content.category"
}
}
},
"aggregations": {
"order_count" : {
"value_count": {
"field": "content.state"
}
}
}
},
"description": "Transform pipeline to put orders into a new index.",
"dest": {
"index": "aggregated_orders001"
},
"settings" : {
"align_checkpoints" : false
},
"frequency": "5m",
"sync": {
"time": {
"field": "latestUpdate",
"delay": "60s"
}
}
}
```
The transform works for both old documents and new incoming documents but not for the updates. So if the state of one document changes to ERROR, the document is still being counted in the transformed index. I thought that this would be the perfect case for using the continous mode with my 'latestUpdate' field but apparently, I do something wrong... I found this "align\_checkpoints" setting and hoped that this would help, but it still doesn't update the counts correctly. Do I maybe need to tell Elastic the time-format of the latestUpdate field, or the time zone?
Also: is there a better way to count the documents in the groups than choosing some field like content.state and do a value\_count?
Any help would be very welcome and appreciated.
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [January 25, 2023, 9:56am UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895/2 "2023-01-25T09:56:50Z")
> [@catrexis](#):
>
> I originally wanted to use 30minutes, but I noticed that only 1 minute or 1 hour is supported but nothing in between, right?
You are using `calendar_interval`, to use 30 minutes you have to use `fixed_interval`, more details in the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-datehistogram-aggregation.html#calendar_and_fixed_intervals).
> [@catrexis](#):
>
> So if the state of one document changes to ERROR, the document is still being counted in the transformed index.
That's because you filtered these documents in the query. The query is applied at the very beginning, making it impossible for transform to take `ERROR` into account.
In order to let transform see the errors, you have to apply the filter as part of aggregations:
```auto
"aggregations": {
"order_count" : {
"filter": {
"bool": {
"must_not":[
{"term":{"content.state":"ERROR"}}
]
}
}
}
}
```
---
### Author:  [@catrexis](https://discuss.elastic.co/u/catrexis)
#### Post date: [January 25, 2023, 12:05pm UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895/3 "2023-01-25T12:05:17Z")
Thank you very much, that worked! 🙂
Do I still need to set align\_checkpoints to false?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [January 25, 2023, 12:32pm UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895/4 "2023-01-25T12:32:07Z")
`align_checkpoints` solves a different use case.
`align_checkpoints` controls when buckets are created. It is a trade-off between recency and resources:
You configured your transform with an interval of `1m`. Assume it is 13:21 and `42s`. With `align_checkpoints:true` transform will create all buckets including `13:20:00`, with `align_checkpoints:false` transform will also create the bucket `13:21:00` although that bucket isn't complete, new data could still arrive. As a consequence transform might update this bucket several times, e.g. at `13:22:42` `true` would write this bucket for the 1st time, `false` would update the bucket. That means `false` is more expensive. However with `align_checkpoints:true` your transformed index lags behind.
With an interval of `1m` this doesn't really matter that much, but think about longer intervals, e.g. `1h`.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 22, 2023, 12:32pm UTC](https://discuss.elastic.co/t/elastic-transforms-continous-mode-is-not-detecting-changes/323895/5 "2023-02-22T12:32:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.