APM Servers hangs on rolling configuration change after a change in elastic

[gmazza (Giovanni Mazza)]

I had applied a Rolling configuration change to my installation.

All Kibana and Elasticsearch nodes came back without a problem but the APM Servers shows as pending configuration.

Also it is showing this alert
APM health issue: Instances are unhealthy

When I review the details for APM i notice on the APM yaml

"system_settings": {
  "elasticsearch_url": "http://a1.containerhost:9244",
  "elasticsearch_username": "cloud-internal-apm-server",
  "kibana_url": "http://containerhost:9244",
  "elasticsearch_password": "password",
  "secret_token": "-"

}

Please advice should that secret_token entry have a value?

i had replaced the specific values for this post but each entry shows values except secret_token which only shows a dash (-)

How I recover from this?

Can i stop the APM server configuration and start fresh?

i really was not doing anything yet with APM servers but I would like APM servers to be available to use.

I can do the APM server deployment again if necessary but not sure how to stop this rolling upgrade for APM and delete the remaining APM nodes ONLY and install new ones without affecting the existing elasticsearch and kibana nodes.

Please advise.

[Yuri (Yuri Tceretian)]

Hi @gmazza, it looks like you use Elastic Search Service (or Elastic Cloud). This forum is for support of the on-premise solution (Elastic Cloud Enterprise). Please file a support ticket in the user console and we will help you fix it.

[gmazza (Giovanni Mazza)]

NO this is on Premise I have version 2.1 and using APM 6.6.1

[gmazza (Giovanni Mazza)]

I stopped the change now it is showing

The requested configuration change was cancelled either in the Cloud UI or by an API call.

You can reapply the configuration change directly.
Please advise what to do to recover

[Yuri (Yuri Tceretian)]

Oh, sorry! That's a known issue in the cloud but I did not think it affected ECE.
The problem here is that APM secret_token got washed out when you applied the plan and that prevented the updated instance to start.
To fix that cluster:

• you need to copy the failed plan and paste to some text editor.
• then open the genesis plan (#0) at the bottom of the activity list and copy value of secert_token.
• replace "-" by the copied secret in the text editor
• Go to advanced APM configuration Link is at the very bottom of page Edit
• Paste the fixed plan and press save button

If it does not help, please contact the support person and they will guide you through this

[gmazza (Giovanni Mazza)]

That worked i copied the secret from the previous into the advance settings and applied and it worked. APM shows now as available.