# \#transforms **URL:** https://discuss.elastic.co/tag/transforms/68.md [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) --- ## [TRANSFORM Destination index is not using Index Template](https://discuss.elastic.co/t/transform-destination-index-is-not-using-index-template/384611)
**Author:** [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Replies:** 1\ **Last updated:** [January 19, 2026, 7:29am UTC](https://discuss.elastic.co/t/transform-destination-index-is-not-using-index-template/384611 "2026-01-19T07:29:20Z")
Hello Team, We are using Elasticsearch 7.8.0 version. I understand its very old \[ w are in process of upgradation too \] While using TRANSFORM feature , I am finding that my destination index is not using template sett… --- ## [Sum the sub buckets returned results in transform](https://discuss.elastic.co/t/sum-the-sub-buckets-returned-results-in-transform/382769)
**Author:** [@Hamza\_Rajput](https://discuss.elastic.co/u/Hamza_Rajput)\ **Replies:** 1\ **Last updated:** [October 16, 2025, 4:24pm UTC](https://discuss.elastic.co/t/sum-the-sub-buckets-returned-results-in-transform/382769 "2025-10-16T16:24:45Z")
Here is my transform preview: POST \_transform/\_preview { "source": { "index": ".ds-metrics-ap.clients-default-\*", "query": { "range": { "@timestamp": { "gte": "now/d", "lt": "… --- ## [Summary index transform reference](https://discuss.elastic.co/t/summary-index-transform-reference/381800)
**Author:** [@bigsby](https://discuss.elastic.co/u/bigsby)\ **Replies:** 1\ **Last updated:** [September 11, 2025, 12:45pm UTC](https://discuss.elastic.co/t/summary-index-transform-reference/381800 "2025-09-11T12:45:54Z")
I have a bunch of indices I’d like merge into a single summary index. From what I read, "Transforming data | Elastic Docs" is the way to go but I’m not find the way to do this. Here it goes. The indices I have are somet… --- ## [Best practice for adding additional fields to transform](https://discuss.elastic.co/t/best-practice-for-adding-additional-fields-to-transform/379665)
**Author:** [@dot-mike](https://discuss.elastic.co/u/dot-mike)\ **Replies:** 2\ **Last updated:** [July 1, 2025, 9:37pm UTC](https://discuss.elastic.co/t/best-practice-for-adding-additional-fields-to-transform/379665 "2025-07-01T21:37:53Z")
Hi community, I've created a custom transform job to extract Fortigate VPN events into custom index to get start / stop time. Sharing it below for reference. This is extracting data from fortigate integration so we alre… --- ## [Best practices for continuous transform with fixed gte filter and tiered storage (hot/warm indices)](https://discuss.elastic.co/t/best-practices-for-continuous-transform-with-fixed-gte-filter-and-tiered-storage-hot-warm-indices/377604)
**Author:** [@ak84](https://discuss.elastic.co/u/ak84)\ **Replies:** 1\ **Last updated:** [April 29, 2025, 6:25pm UTC](https://discuss.elastic.co/t/best-practices-for-continuous-transform-with-fixed-gte-filter-and-tiered-storage-hot-warm-indices/377604 "2025-04-29T18:25:00Z")
I'm using an Elasticsearch continuous transform to aggregate data from time-based indices (These are daily indices which contain time sensitive documents). My goal is to have the transform only process data starting from… --- ## [Indexing rate for data streams](https://discuss.elastic.co/t/indexing-rate-for-data-streams/376566)
**Author:** [@Dr00py](https://discuss.elastic.co/u/Dr00py)\ **Replies:** 3\ **Last updated:** [April 2, 2025, 4:50pm UTC](https://discuss.elastic.co/t/indexing-rate-for-data-streams/376566 "2025-04-02T16:50:17Z")
I want to calculate the indexing speed of documents in primary shards in my data streams. When monitoring is enabled, I can see the index rate for each index. But I can't do this for any data stream from my clusters. I … --- ## [Get the latest data for each day](https://discuss.elastic.co/t/get-the-latest-data-for-each-day/376221)
**Author:** [@Eloise\_Wilkinson](https://discuss.elastic.co/u/Eloise_Wilkinson)\ **Replies:** 6\ **Last updated:** [March 21, 2025, 4:30pm UTC](https://discuss.elastic.co/t/get-the-latest-data-for-each-day/376221 "2025-03-21T16:30:56Z")
Hi, The goal I am trying to achieve is to display the latest status of my data per day. So for example, I want to have the latest data for the latest timestamp on Monday, Tuesday, Wednesday etc. I tried to achieve this … --- ## [TooManyBucketsException in Transform with Continuous High Ingestion (100k documents/minute)](https://discuss.elastic.co/t/toomanybucketsexception-in-transform-with-continuous-high-ingestion-100k-documents-minute/375509)
**Author:** [@Lucas\_Pereira](https://discuss.elastic.co/u/Lucas_Pereira)\ **Replies:** 4\ **Last updated:** [March 19, 2025, 3:23pm UTC](https://discuss.elastic.co/t/toomanybucketsexception-in-transform-with-continuous-high-ingestion-100k-documents-minute/375509 "2025-03-19T15:23:35Z")
Hi, I'm facing an issue with an Elasticsearch transform that can't handle a continuous stream of high ingestion (~100k documents per minute) into the netflow-read-\* source index, as well as occasional backlogs (e.g. 10M … --- ## [No doc count in destination index of transform](https://discuss.elastic.co/t/no-doc-count-in-destination-index-of-transform/373284)
**Author:** [@sintim](https://discuss.elastic.co/u/sintim)\ **Replies:** 3\ **Last updated:** [January 17, 2025, 2:00pm UTC](https://discuss.elastic.co/t/no-doc-count-in-destination-index-of-transform/373284 "2025-01-17T14:00:27Z")
I created a transform on ELK and it is up and HEALTHY but till now there's still no documents in the destination index for a data view. It's been about 6 hours. Is that supposed to be normal? or if not how can I check if… --- ## [Combine connect and disconnect documents with transform?](https://discuss.elastic.co/t/combine-connect-and-disconnect-documents-with-transform/372477)
**Author:** [@Jokke](https://discuss.elastic.co/u/Jokke)\ **Replies:** 3\ **Last updated:** [January 9, 2025, 1:29pm UTC](https://discuss.elastic.co/t/combine-connect-and-disconnect-documents-with-transform/372477 "2025-01-09T13:29:02Z")
Hello! I'm new to transforms so not really sure, but I think transforms is what I need here. Logs are pushed from web application to ES (fluentd) and there among connect and disconnect messages. What I would want is to… --- ## [Transforming (aggregating) network logs 14 million records every 5 min](https://discuss.elastic.co/t/transforming-aggregating-network-logs-14-million-records-every-5-min/371560)
**Author:** [@sid\_shah](https://discuss.elastic.co/u/sid_shah)\ **Replies:** 8\ **Last updated:** [December 7, 2024, 2:39pm UTC](https://discuss.elastic.co/t/transforming-aggregating-network-logs-14-million-records-every-5-min/371560 "2024-12-07T14:39:33Z")
I have 14 million records arriving every 5 minutes from Filebeat into a data stream (NetFlow data). My goal is to aggregate this data based on certain rules, such as summing source bytes and destination bytes for sp… --- ## [How to calculate the standard deviation in a transform?](https://discuss.elastic.co/t/how-to-calculate-the-standard-deviation-in-a-transform/368845)
**Author:** [@kishorkumar](https://discuss.elastic.co/u/kishorkumar)\ **Replies:** 4\ **Last updated:** [October 17, 2024, 1:05pm UTC](https://discuss.elastic.co/t/how-to-calculate-the-standard-deviation-in-a-transform/368845 "2024-10-17T13:05:59Z")
I am trying to create a transform report, but I'm unable to create a range based on the avg(total). For example, I want to assign customers into a bucket based on their average total spend, like the range $0-$100. Can … --- ## [Transform with two input indices with different unique ids](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475)
**Author:** [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Replies:** 3\ **Last updated:** [October 16, 2024, 9:11pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475 "2024-10-16T21:11:18Z")
Hi, we have a transform with two input indices with different unique ids. One input index has a unique id of product\_pk, and another has product\_pk combined with catalog\_type. There are two possible catalog\_types: cata… --- ## [Does sort need to be applied to query for applying index-sort optimizations? What about transforms?](https://discuss.elastic.co/t/does-sort-need-to-be-applied-to-query-for-applying-index-sort-optimizations-what-about-transforms/368872)
**Author:** [@Max5](https://discuss.elastic.co/u/Max5)\ **Replies:** 0\ **Last updated:** [October 15, 2024, 8:00pm UTC](https://discuss.elastic.co/t/does-sort-need-to-be-applied-to-query-for-applying-index-sort-optimizations-what-about-transforms/368872 "2024-10-15T20:00:46Z")
I'm trying to fully understand the best way to optimize for transforms and then searching running aggregations on resulting indices. The situation: Stuck on 7.17.21 currently Non-managed cluster Conversion to data str… --- ## [Retention errors in Transforms](https://discuss.elastic.co/t/retention-errors-in-transforms/368747)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 1\ **Last updated:** [October 15, 2024, 2:45pm UTC](https://discuss.elastic.co/t/retention-errors-in-transforms/368747 "2024-10-15T14:45:19Z")
Hi Team, We are using transforms in our cluster and we have applied retention in the transforms for 90 days. But now a days we are frequently seeing below warnings in transforms. Could you please help me to understand t… --- ## [Join two indices (many-to-many relationships)](https://discuss.elastic.co/t/join-two-indices-many-to-many-relationships/363867)
**Author:** [@bianca\_s](https://discuss.elastic.co/u/bianca_s)\ **Replies:** 2\ **Last updated:** [October 14, 2024, 7:48am UTC](https://discuss.elastic.co/t/join-two-indices-many-to-many-relationships/363867 "2024-10-14T07:48:23Z")
Hi all, we have the following use case: We want to create an index that holds the relationships of producer - topic - consumer. Basically all relationships here are many-to-many: Multiple producers can write to the s… --- ## [Latest unique transform missing documents](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700)
**Author:** [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Replies:** 1\ **Last updated:** [October 11, 2024, 9:18pm UTC](https://discuss.elastic.co/t/latest-unique-transform-missing-documents/368700 "2024-10-11T21:18:05Z")
We have what I believe to be a straightforward transform to pick out unique documents ordered by an ingest pipeline timestamp. We apply it to several programs (customers) index data, but with larger data it is missing d… --- ## [Transform job status remains yellow](https://discuss.elastic.co/t/transform-job-status-remains-yellow/368094)
**Author:** [@gueri](https://discuss.elastic.co/u/gueri)\ **Replies:** 1\ **Last updated:** [October 2, 2024, 3:02pm UTC](https://discuss.elastic.co/t/transform-job-status-remains-yellow/368094 "2024-10-02T15:02:23Z")
Hello, It seems our cluster encountered an error this morning that impacted my transform job in continuous mode Transform encountered an exception: \[Search rejected due to missing shards. I don't know exactly what ha… --- ## [Sorting the data based on date in Elasticseach using the transform](https://discuss.elastic.co/t/sorting-the-data-based-on-date-in-elasticseach-using-the-transform/366734)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 2\ **Last updated:** [September 18, 2024, 2:19pm UTC](https://discuss.elastic.co/t/sorting-the-data-based-on-date-in-elasticseach-using-the-transform/366734 "2024-09-18T14:19:29Z")
Hi Team, We are using a sort in our transform to sort the records based on a timestamp field by grouping them on a unique key and get only the latest record out of that. But we are facing an issue as in one use case my … --- ## [Can we use ILM on the index created by using transform](https://discuss.elastic.co/t/can-we-use-ilm-on-the-index-created-by-using-transform/366445)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 1\ **Last updated:** [September 12, 2024, 6:15am UTC](https://discuss.elastic.co/t/can-we-use-ilm-on-the-index-created-by-using-transform/366445 "2024-09-12T06:15:42Z")
Hi Team, we are using 180 days retention for all the data in our Elastic cluster. Now we have some transforms where we are using the transforms retention\_policy to keep the data for 180 days. Now we wanted to remove the… --- ## [Elasticsearch transformation query](https://discuss.elastic.co/t/elasticsearch-transformation-query/366147)
**Author:** [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\ **Replies:** 1\ **Last updated:** [September 6, 2024, 1:22pm UTC](https://discuss.elastic.co/t/elasticsearch-transformation-query/366147 "2024-09-06T13:22:39Z")
We are working on a data processing pipeline that involves multiple transformations. Specifically, we have a use case where the first transformation runs and calculates documents for various systems, including system1. I… --- ## [Missing data in daily aggregate compared to source and 5min aggregate](https://discuss.elastic.co/t/missing-data-in-daily-aggregate-compared-to-source-and-5min-aggregate/364468)
**Author:** [@Cobraeti](https://discuss.elastic.co/u/Cobraeti)\ **Replies:** 7\ **Last updated:** [September 3, 2024, 2:24pm UTC](https://discuss.elastic.co/t/missing-data-in-daily-aggregate-compared-to-source-and-5min-aggregate/364468 "2024-09-03T14:24:53Z")
Hello, We noticed a miss-match in some daily aggregates compared to 5min aggregates based on the same source data: As you can see on the above charts, the 1day\_agg chart shows one day with a way lower value (not 0 t… --- ## [Using Transform for document count when document updated](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924)
**Author:** [@Derek.X](https://discuss.elastic.co/u/Derek.X)\ **Replies:** 3\ **Last updated:** [August 27, 2024, 2:44pm UTC](https://discuss.elastic.co/t/using-transform-for-document-count-when-document-updated/364924 "2024-08-27T14:44:02Z")
There is an index that documents are updated with time. We are looking for some way to continuesly (every several minutes) provide count of documents grouped by some condition. Example: 2024/08/15 00:00:00, order1, new … --- ## [Understanding operations\_behind in transform stats](https://discuss.elastic.co/t/understanding-operations-behind-in-transform-stats/363281)
**Author:** [@gueri](https://discuss.elastic.co/u/gueri)\ **Replies:** 2\ **Last updated:** [August 6, 2024, 2:10pm UTC](https://discuss.elastic.co/t/understanding-operations-behind-in-transform-stats/363281 "2024-08-06T14:10:55Z")
Hello, I have a question about the metric operations\_behind when I call the transform stats API. The elastic doc says the following about operations\_behind : The number of operations that have occurred on the source … --- ## [Transform results in index with missing documents when using the API but works for console](https://discuss.elastic.co/t/transform-results-in-index-with-missing-documents-when-using-the-api-but-works-for-console/363044)
**Author:** [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Replies:** 13\ **Last updated:** [July 19, 2024, 6:38pm UTC](https://discuss.elastic.co/t/transform-results-in-index-with-missing-documents-when-using-the-api-but-works-for-console/363044 "2024-07-19T18:38:22Z")
Hi. We have developed a transform that works given the same input indices in the console. However, when using the golang TransformPutTransform API, many of the documents are missing in the resulting index. Here is the… --- ## [Error while running transform \`task encountered irrecoverable failure\`](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479)
**Author:** [@lizozom](https://discuss.elastic.co/u/lizozom)\ **Replies:** 5\ **Last updated:** [July 12, 2024, 4:05pm UTC](https://discuss.elastic.co/t/error-while-running-transform-task-encountered-irrecoverable-failure/362479 "2024-07-12T16:05:57Z")
Hey team, I'm running ELK 8.13 I have a transform that occasionally fails to run. If I restart it - the failure persists. If I recreate it - the error goes away for a few days and that returns. It fails with this er… --- ## [Impact of frequency value for continuous transform](https://discuss.elastic.co/t/impact-of-frequency-value-for-continuous-transform/362903)
**Author:** [@gueri](https://discuss.elastic.co/u/gueri)\ **Replies:** 3\ **Last updated:** [July 11, 2024, 10:25pm UTC](https://discuss.elastic.co/t/impact-of-frequency-value-for-continuous-transform/362903 "2024-07-11T22:25:28Z")
Hi, I created a continuous transform job to run once a day and calculate aggregations about communications with IPs and protocole/port. This is the skeleton of my transform { "source": { "index": \[ "my\_sou… --- ## [How to Display Results in Table that are "Intersection" of two Queries?](https://discuss.elastic.co/t/how-to-display-results-in-table-that-are-intersection-of-two-queries/360978)
**Author:** [@bianca\_s](https://discuss.elastic.co/u/bianca_s)\ **Replies:** 3\ **Last updated:** [June 27, 2024, 11:56am UTC](https://discuss.elastic.co/t/how-to-display-results-in-table-that-are-intersection-of-two-queries/360978 "2024-06-27T11:56:55Z")
Hi all, we have the following use case: We have an index containing logs of authentications from different devices. The data looks something like this: device\_id region\_id district\_id 1234 A district-1 5678 A… --- ## [Can't start any pivot transform - getting allocation explanation error without reason](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849)
**Author:** [@Django](https://discuss.elastic.co/u/Django)\ **Replies:** 4\ **Last updated:** [June 6, 2024, 9:21am UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849 "2024-06-06T09:21:09Z")
Hi! Something strange happened with our Elasticsearch v7.17 cluster - I can't start pivot transforms due to an error {"root\_cause":\[{"type":"status\_exception","reason":"Could not start transform, allocation explanation… --- ## [Continuous transform of a transform destination index](https://discuss.elastic.co/t/continuous-transform-of-a-transform-destination-index/359577)
**Author:** [@vstokarev](https://discuss.elastic.co/u/vstokarev)\ **Replies:** 1\ **Last updated:** [May 16, 2024, 9:48pm UTC](https://discuss.elastic.co/t/continuous-transform-of-a-transform-destination-index/359577 "2024-05-16T21:48:22Z")
I need to create a transform that will process the data from another transform's destination index to further aggregate the aggregated data. It works fine as a one-time job, but can it work continuously? Considering tha… [Next page](https://discuss.elastic.co/tag/transforms/68.md?match_all_tags=true&page=1&tags%5B%5D=transforms)