# Using Transform for document count when document updated
**URL:** https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924
**Category:** Elasticsearch
**Tags:** transforms
**Created:** [August 14, 2024, 5:31pm UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924 "2024-08-14T17:31:03Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Derek.X](https://discuss.elastic.co/u/Derek.X)
#### Post date: [August 14, 2024, 5:31pm UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924/1 "2024-08-14T17:31:03Z")
There is an index that documents are updated with time. We are looking for some way to continuesly (every several minutes) provide count of documents grouped by some condition.
Example:
2024/08/15 00:00:00, order1, new
2024/08/15 01:00:00, order2, payed
2024/08/15 00:00:00, order3, payed
one our later, the index may looks like
2024/08/15 01:15:00, order1, payed
2024/08/15 01:00:00, order2, payed
2024/08/15 01:20:00, order3, shipped
2024/08/15 01:30:00, order4, payed
We are thinking to implement it with Elasticsearch transform, however in above senario, the count of "new" orders may remain as one. We are also thinking about other alternatives, such as add a timestamp indictate when transform generated document is last updated, but not sure if it is possible.
The queries are:
Is the above understanding is correct - Transform will not delete {status : "new",count : "1"} from transform dest index in above senario?
If this is true, could you suggest if there is any work around, like add an field indicating when the transform result is generated/updated
Thank you in advance
This query may similar to [Transform behavior with deleted documents](https://discuss.elastic.co/t/transform-behavior-with-deleted-documents/240172)
---
### Author:  [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)
#### Post date: [August 14, 2024, 6:48pm UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924/2 "2024-08-14T18:48:53Z")
> Is the above understanding is correct - transform will not delete {status : "new",count : "1"} from transform dest index in above senario?
This is correct.
> like add an field indicating when the transform result is generated/updated
This other answer may help - [Creating an ingest pipeline for transforms](https://discuss.elastic.co/t/transform-results-in-index-with-missing-documents-when-using-the-api-but-works-for-console/363044/9).
- create an ingest pipeline that adds a timestamp field
- use the [Put Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) API to set the pipeline in the `dest.pipeline` field
Any document added or updated in the destination index will get a new timestamp.
It would look something like:
```auto
PUT _ingest/pipeline/pipeline_add_ingest_timestamp
{
"description": "Adds event.ingested field which represents time of ingestion.",
"processors": [
{
"set": {
"field": "event.ingested",
"value": "{{_ingest.timestamp}}"
}
}
]
}
```
```auto
PUT _transform/new_orders_count
{
...
"dest": {
"index": "dest_index_name",
"pipeline": "pipeline_add_ingest_timestamp"
},
...
}
```
---
### Author:  [@Derek.X](https://discuss.elastic.co/u/Derek.X)
#### Post date: [August 23, 2024, 7:24am UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924/3 "2024-08-23T07:24:44Z")
Thank you for the reply. We have setup to the transform and ingest pipeline, and we do see a ingest event timestamp added to docs.
Actually we are expected the ingest event will tell us what are transformed records are latest(different from latest updated), so that reader will be able to filter out items based on ingest event (to get docs expected in original post). However, it seems only latest updated docs have ingest event updated.
So seems transform does not update those records that having the same transform result, and ingest event time not updated.
Example:
2024/08/15 00:00:00, order1, new
2024/08/15 00:50:00, order2, payed
one our later, the index may looks like
2024/08/15 00:00:00, order1, new
2024/08/15 01:15:00, order2, closed
Before:
```auto
{status : "new", count : "1", ingest_time: "1:00:00"}
{status : "payed", count : "1",ingest_time: "1:00:00"}
```
After Actual
```auto
{status : "new", count : "1", ingest_time: "1:00:00"} --2 00 00 expected
{status : "closed", count : "1",ingest_time: "2:00:00"}
{status : "payed", count : "1", ingest_time: "1:00:00"}
```
If this is correct, I am thinking we have to turn to periodical elasticsearch query to get statistic.
---
### Author:  [@Derek.X](https://discuss.elastic.co/u/Derek.X)
#### Post date: [August 27, 2024, 2:44pm UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924/4 "2024-08-27T14:44:02Z")
Checked multiple other potential solutions, I am trying to exporter the aggregation data to some other data storage.
Logstash elasticsearch input plugin seems to be an alternative. Sadly plugin version we are using does no support it.
>
>
> As title says, this PR enables this plugin to execute aggregation queries
>
> The… work is based on the existing PR #90 (and potentially resolves #58)
> However code & tests were updated to include all the changes that were introduced since original PR was created/last updated
>
> Tests are included and also was live tested with Logstash v8.8.2
Watcher seems to be another alternative. Below is a thread about it
> [@Watcher condition with aggregation](https://discuss.elastic.co/t/watcher-condition-with-aggregation/225758/5):
>
> Hi @cjcenizal. Could you provide any feedback on my results? slight_smile Also, would array compare be applicable here? "condition": { "array\_compare": { "ctx.payload.aggregations.node\_name.buckets" : { "path": "unsure which path leads to unack\_count.value", "gte": { "value": 0 } } } }