# Can't start any pivot transform - getting allocation explanation error without reason

**URL:** <https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849>\
**Category:** Elasticsearch\
**Tags:** transforms\
**Created:** [June 5, 2024, 12:33pm UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849 "2024-06-05T12:33:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Django](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@Django](https://discuss.elastic.co/u/Django)\
**Post date:** [June 5, 2024, 12:33pm UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849/1 "2024-06-05T12:33:00Z")

</div>

Hi!

Something strange happened with our Elasticsearch v7.17 cluster - I can't start pivot transforms due to an error  
`{"root_cause":[{"type":"status_exception","reason":"Could not start transform, allocation explanation [Not starting transform [test-transform], reasons []]"}],"type":"status_exception","reason":"Could not start transform, allocation explanation [Not starting transform [test-transform], reasons []]"}`

I tried to create a few pivot transforms, with different sets of parameters and different sources, but always got the same error. I guess, this happened after I removed the oldest and weakest Elasticsearch node from cluster. I did this because for unclear reasons, transforms always ran on that node, despite that in cluster available a few much more powerful nodes with "transform" role. Before removing that old node from the cluster, I migrated data to other nodes, then used node shutdown API with "type": "remove" option. Later, I tried to return the old node to the cluster, but this didn't help.  
I checked Elasticsearch logs on nodes but found nothing useful. Also, I checked common cluster issues, described in [Fix common cluster issues | Elasticsearch Guide [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/fix-common-cluster-issues.html), but no obvious bottleneck was found.  
I'm stuck and have no idea how to debug this further.

Thanks for your attention!

---

<div class="post-metadata">

**Author:** ![greco](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/greco/32/106815_2.png) [@greco](https://discuss.elastic.co/u/greco)\
**Post date:** [June 5, 2024, 1:51pm UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849/2 "2024-06-05T13:51:35Z")

</div>

Hi @Django,

Can you check if one of your node has the transform role ? You can do this by running in the Dev console :

> GET \_cat/nodes

and checking if a node has the letter "t" in its roles.

If that's not the case, you should probably configure a node to be a transform node. Check [this page](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#transform-node) of our documentation to do so.

Hope that helps

---

<div class="post-metadata">

**Author:** ![Django](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@Django](https://discuss.elastic.co/u/Django)\
**Post date:** [June 5, 2024, 2:03pm UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849/3 "2024-06-05T14:03:32Z")

</div>

Thanks for the response, @greco ! They all have this role.

```auto
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
139.x.x.x 15 99 0 0.05 0.05 0.07 hirstw - elk8
139.x.x.x 58 99 18 4.50 4.85 4.86 chimrstw * elk5
138.x.x.x 46 96 5 2.75 2.94 4.75 hirstw - elk7
130.x.x.x 50 90 0 0.41 0.23 0.29 himrstvw - elk3
139.x.x.x 37 100 7 2.46 2.66 2.83 himrstw - elk6

```

IDK why, but before I removed oldest node from the cluster, transforms complained on start "Could not start transform, allocation explanation [Not starting transform [test-transform], reasons [node\_id:not a transform node]]"},"status":429} - and that was for every node in cluster, that doesn't have "transform" role yet. So, I forced to assign this role to all nodes.

---

<div class="post-metadata">

**Author:** ![Patrick\_Whelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_whelan/32/135049_2.png) [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)\
**Post date:** [June 5, 2024, 2:22pm UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849/4 "2024-06-05T14:22:38Z")

</div>

From the [transforms code](https://github.com/elastic/elasticsearch/blob/dc13b75656146b59d5e52be7fd78b288237cf7bd/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/transforms/TransformPersistentTasksExecutor.java#L145), it seems `reasons[]` would be empty if it cannot find any nodes in the cluster.

Going off of [this guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-troubleshooting.html), are there any errors or anything in the logs that indicate the cluster in unstable?

---

<div class="post-metadata">

**Author:** ![Django](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@Django](https://discuss.elastic.co/u/Django)\
**Post date:** [June 6, 2024, 9:21am UTC](https://discuss.elastic.co/t/cant-start-any-pivot-transform-getting-allocation-explanation-error-without-reason/360849/5 "2024-06-06T09:21:09Z")

</div>

> [@Patrick\_Whelan](#):
>
> are there any errors or anything in the logs that indicate the cluster in unstable?

Thank you, @Patrick_Whelan for advice! Unfortunately, I found no signs of cluster instability - no repeated master elections (except when I restarted the master), no node join failures, and no flapping node connections... Your advice prompted me to check "discovery.seed\_hosts" lists in nodes' elasticsearch.yml - seed\_hosts indeed were partially outdated. I deleted from the "discovery.seed\_hosts" non-existed any more nodes, updated them to the current list of nodes, and restarted nodes accordingly - nothing changed, test transform still can't start, the error is the same - empty `reasons[]`. Are there any other ways to troubleshoot this transform failure?
