# Problem creating transforms
**URL:** https://discuss.elastic.co/t/problem-creating-transforms/210701
**Category:** Elasticsearch
**Tags:** elastic-stack-machine-learning
**Created:** [December 5, 2019, 12:15pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701 "2019-12-05T12:15:12Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 5, 2019, 12:15pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/1 "2019-12-05T12:15:12Z")
Hi,
I upgraded by cluster to version 7.5 and I'm trying the Transforms from Kibana but I'm having some problem that I cannot identify.
The big picture is that when I create a continuous transform with more than 4-5 aggregations, the transform is created but not the index and it can't be started. When it happens and I click on the "Start" button, the only message I receive is:
"An error occurred starting the transform XXX"
with no further detail...
I've tried creating exactly the same transform (same configuration) and changing the number of aggregations: 1, 2, 3... My final goal is creating one with 13 aggregations but I could never create any transform with more than 4 aggregations.
I also checked that is not a problem with the fields I'm aggregating, because for example, if I create 1 transform with:
- avg(field1)
- sum(field1)
- avg(field2)
- sum(field2)
- avg(field3)
- sum(field3)
- avg(field4)
- sum(field4)
It always fails, but if I do 2 transforms with the same aggregations splitted into fields1-2 and fields3-4, then both transforms works normally.
Is there some limitations in the number of transforms supported?
Note: I'm using Elastic Service
Thanks in advance!!!
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 5, 2019, 1:01pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/2 "2019-12-05T13:01:33Z")
Hi,
there is no limitation in the number of aggregations. Therefore it seems your transform fails for other reasons.
The destination index is created on start, not as part of creating the transform. On start transform checks whether the destination index exists and creates it if not. It would be good to know if the destination index gets created or not.
To further debug your issue:
Can you check the message tab of your job, you can find it in the jobs list after expanding the view, for example:
In addition it would be useful if you can query for transform stats in the dev console like this:
```auto
GET _transform/{your_failing_transform}/_stats
```
The output of this command should contain the failure reason.
Please let me know about your findings.
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 5, 2019, 3:20pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/3 "2019-12-05T15:20:17Z")
Hi Hendrik,
Thanks for your reply.
The index never is created and I double checked that there is no index with that name as you can see in the following images:
So, I guess the problem is at index creation, but I can't figure out the reason because there is no error message:
Is there anything else I can check to find the error?
Thanks!
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 5, 2019, 7:12pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/4 "2019-12-05T19:12:29Z")
Hi,
the error happens when transforms creates the index (the audit message is send right before the index creation).
Can you start the transform via the dev console and send me the output:
```auto
POST _transform/orders_by_cutomers_min/_start
```
The full error should be in the response.
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 6, 2019, 8:11am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/5 "2019-12-06T08:11:45Z")
Hi Hendrik,
This is the error from the Dev console:
I checked the mapping of that field and it is a float field:
In fact, during my tests, I created other continuous transforms with the same aggregations for that field and they work perfectly:
Further, the Preview of the transform that is failling does also work:
Any idea?
Many thanks!
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 6, 2019, 8:42am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/6 "2019-12-06T08:42:10Z")
Good morning!
We are getting closer, thanks for your patience.
The error means in a nutshell that transform tries to create a destination index with a field `subtotal_price` of some simple type and a nested field `subtotal_price.something`, so basically an object with another field in it.
I tried to reproduce your problem: for me the UI errors for such a mapping conflict on preview. I would like to find out, why for you it does not, seems like a special case.
Can you share your configuration, e.g. for the preview, you find a copy method on the upper right corner of your preview in the UI:
FWIW: As a workaround you can rename all your nested fields, e.g `subtotal_price.avg` to `subtotal_price_avg`.
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 6, 2019, 9:39am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/7 "2019-12-06T09:39:23Z")
Hi Hendrik,
Thank you for your patient and your time.
Here is the configuration os the "source index preview":
```
GET orders/_search
{
"query": {
"bool": {
"must_not": {
"bool": {
"should": [
{
"exists": {
"field": "cancelled_at"
}
}
],
"minimum_should_match": 1
}
}
}
}
}
```
And, this is the full transform:
```
POST _transform/_preview
{
"source": {
"index": [
"orders"
],
"query": {
"bool": {
"must_not": {
"bool": {
"should": [
{
"exists": {
"field": "cancelled_at"
}
}
],
"minimum_should_match": 1
}
}
}
}
},
"pivot": {
"group_by": {
"customer.email.keyword": {
"terms": {
"field": "customer.email.keyword"
}
}
},
"aggregations": {
"subtotal_price.avg": {
"avg": {
"field": "subtotal_price"
}
},
"subtotal_price.sum": {
"sum": {
"field": "subtotal_price"
}
},
"total_discounts.avg": {
"avg": {
"field": "total_discounts"
}
},
"total_discounts.sum": {
"sum": {
"field": "total_discounts"
}
},
"total_price.avg": {
"avg": {
"field": "total_price"
}
},
"total_price.sum": {
"sum": {
"field": "total_price"
}
},
"customer.orders_count.max": {
"max": {
"field": "customer.orders_count"
}
}
}
}
}
```
Cheers
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 6, 2019, 10:12am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/8 "2019-12-06T10:12:13Z")
Hi,
can you send me the output of the preview call, after you executed it in the dev console? It returns the mappings it would use to create the destination index.
I see nothing wrong with your config. This is a transform that fails on start for you?
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 6, 2019, 10:30am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/9 "2019-12-06T10:30:05Z")
Sure, here it is:
```
{
"preview" : [
{
"total_discounts" : {
"avg" : 0.0,
"sum" : 0.0
},
"subtotal_price" : {
"avg" : 323.0,
"sum" : 323.0
},
"total_price" : {
"avg" : 323.0,
"sum" : 323.0
},
"customer" : {
"orders_count" : {
"max" : 1.0
},
"email" : {
"keyword" : "XXXXX"
}
}
},
{...rest of the results...}
],
"mappings" : {
"properties" : {
"total_discounts.sum" : {
"type" : "double"
},
"subtotal_price.sum" : {
"type" : "double"
},
"total_price.sum" : {
"type" : "double"
},
"total_price.avg" : {
"type" : "double"
},
"customer.email.keyword" : {
"type" : "keyword"
},
"total_discounts.avg" : {
"type" : "double"
},
"customer.orders_count.max" : {
"type" : "long"
},
"subtotal_price.avg" : {
"type" : "double"
}
}
}
}
```
Many thanks!!
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 6, 2019, 11:15am UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/10 "2019-12-06T11:15:45Z")
Hi,
the mappings look fine, however there might be another source for conflicting mappings: index templates.
To find out, you can try to create your problematic transform with a destination index not starting with the prefix `orders`, for example `transform_orders_by_customers_min`?
If that still fails and/or to check whether the suspicion is correct, can you get the index templates with:
```auto
GET /_template
```
and look for templates that start with `orders` or a template that would match `orders`, e.g. a template for `*`? The pattern is defined in the field `index_patterns` of the individual templates.
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 6, 2019, 12:28pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/11 "2019-12-06T12:28:26Z")
Your are great, Hendrik!! That was the problem.
There is one template that is affecting the final mapping, although I don't know exactly why.
I checked all my templates and find the only one that matches the index name and review the mapping and it looks OK to me (I keep just the fields that I'm using for grouping and aggregating):
```
[...]
"index_patterns" : [
"orders*"
],
[...]
"mappings" : {
"properties" : {
[...]
"subtotal_price" : {
"type" : "float"
},
"total_discounts" : {
"type" : "float"
},
"total_price" : {
"type" : "float"
},
"customer" : {
"properties" : {
"orders_count" : {
"type" : "long"
},
"email" : {
"type" : "text",
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
}
},
[...]
},
[...]
```
If you know the reason, please share it with me.
In any way, thanks a lot for helping me with this!!!
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [December 6, 2019, 12:48pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/12 "2019-12-06T12:48:34Z")
The problem is that the template defines:
```auto
{
"subtotal_price" : {
"type" : "float"
},
}
```
which in json would be for example:
```auto
{
"subtotal_price": 55
}
```
but in the transform you create:
```auto
{
"subtotal_price.sum" : {
"type" : "double"
},
"subtotal_price.avg" : {
"type" : "double"
}
}
```
which in json looks like this:
```auto
{
"subtotal_price": {
"sum": 100,
"avg": 23
}
}
```
As you can see `sum` and `avg` are nested objects of `subtotal_price`. You can not have something that is a number and an object at the same time.
I admit the error message is quite cryptic and it would be nice if request validation would have found this problem before `_start`.
Several usability issues have been found thanks to you. I will follow up and put them on the backlog for upcoming releases.
---
### Author:  [@JulGor](https://discuss.elastic.co/u/JulGor)
#### Post date: [December 6, 2019, 9:54pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/13 "2019-12-06T21:54:40Z")
Now I got it.
Thanks again for your help and your time!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 3, 2020, 9:54pm UTC](https://discuss.elastic.co/t/problem-creating-transforms/210701/14 "2020-01-03T21:54:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.