# APM Servers hangs on rolling configuration change after a change in elastic **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Created:** [March 5, 2019, 8:19pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985 "2019-03-05T20:19:44Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![gmazza](https://avatars.discourse-cdn.com/v4/letter/g/82dd89/32.png) [@gmazza](https://discuss.elastic.co/u/gmazza)\ **Post date:** [March 5, 2019, 8:19pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/1 "2019-03-05T20:19:44Z")
I had applied a Rolling configuration change to my installation. All Kibana and Elasticsearch nodes came back without a problem but the APM Servers shows as pending configuration. Also it is showing this alert APM health issue: Instances are unhealthy When I review the details for APM i notice on the APM yaml ``` "system_settings": { "elasticsearch_url": "http://a1.containerhost:9244", "elasticsearch_username": "cloud-internal-apm-server", "kibana_url": "http://containerhost:9244", "elasticsearch_password": "password", "secret_token": "-" ``` } Please advice should that secret\_token entry have a value? i had replaced the specific values for this post but each entry shows values except secret\_token which only shows a dash (-) How I recover from this? Can i stop the APM server configuration and start fresh? i really was not doing anything yet with APM servers but I would like APM servers to be available to use. I can do the APM server deployment again if necessary but not sure how to stop this rolling upgrade for APM and delete the remaining APM nodes ONLY and install new ones without affecting the existing elasticsearch and kibana nodes. Please advise. ---
**Author:** ![Yuri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuri/32/52458_2.png) [@Yuri](https://discuss.elastic.co/u/Yuri)\ **Post date:** [March 5, 2019, 8:37pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/2 "2019-03-05T20:37:06Z")
Hi @gmazza, it looks like you use Elastic Search Service (or Elastic Cloud). This forum is for support of the on-premise solution (Elastic Cloud Enterprise). Please file a support ticket in the user console and we will help you fix it. ---
**Author:** ![Yuri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuri/32/52458_2.png) [@Yuri](https://discuss.elastic.co/u/Yuri)\ **Post date:** [March 5, 2019, 8:37pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/3 "2019-03-05T20:37:14Z")
---
**Author:** ![gmazza](https://avatars.discourse-cdn.com/v4/letter/g/82dd89/32.png) [@gmazza](https://discuss.elastic.co/u/gmazza)\ **Post date:** [March 5, 2019, 8:38pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/4 "2019-03-05T20:38:00Z")
NO this is on Premise I have version 2.1 and using APM 6.6.1 ---
**Author:** ![gmazza](https://avatars.discourse-cdn.com/v4/letter/g/82dd89/32.png) [@gmazza](https://discuss.elastic.co/u/gmazza)\ **Post date:** [March 5, 2019, 8:39pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/5 "2019-03-05T20:39:20Z")
I stopped the change now it is showing The requested configuration change was cancelled either in the Cloud UI or by an API call. You can reapply the configuration change directly. Please advise what to do to recover ---
**Author:** ![Yuri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuri/32/52458_2.png) [@Yuri](https://discuss.elastic.co/u/Yuri)\ **Post date:** [March 5, 2019, 8:41pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/6 "2019-03-05T20:41:23Z")
---
**Author:** ![Yuri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuri/32/52458_2.png) [@Yuri](https://discuss.elastic.co/u/Yuri)\ **Post date:** [March 5, 2019, 8:52pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/7 "2019-03-05T20:52:34Z")
Oh, sorry! That's a known issue in the cloud but I did not think it affected ECE. The problem here is that APM `secret_token` got washed out when you applied the plan and that prevented the updated instance to start. To fix that cluster: - you need to copy the failed plan and paste to some text editor. - then open the genesis plan (`#0`) at the bottom of the activity list and copy value of `secert_token`. - replace `"-"` by the copied secret in the text editor - Go to `advanced APM configuration` Link is at the very bottom of page `Edit` - Paste the fixed plan and press save button If it does not help, please contact the support person and they will guide you through this ---
**Author:** ![gmazza](https://avatars.discourse-cdn.com/v4/letter/g/82dd89/32.png) [@gmazza](https://discuss.elastic.co/u/gmazza)\ **Post date:** [March 5, 2019, 9:43pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/8 "2019-03-05T21:43:43Z")
That worked i copied the secret from the previous into the advance settings and applied and it worked. APM shows now as available. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 19, 2019, 9:43pm UTC](https://discuss.elastic.co/t/apm-servers-hangs-on-rolling-configuration-change-after-a-change-in-elastic/170985/9 "2019-03-19T21:43:47Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.