Hi Martin,
Unfortunately, after upgrading, some agents can get stuck in "updating" state in Kibana, even if the actual underlying state is "healthy".
To un-mark any Elastic Agent as upgrading, you can follow the below commands as superuser.
Create a new service token:
curl -X POST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H' content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"
Next, do an update_by_query
. The query finds all Elastic Agents which are marked updating
and marks the update as complete. With the created token replace the ${TOKEN}
below with that created token so the query can complete successfully:
curl -XPOST -H' Authorization: Bearer ${TOKEN}' -H' x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "upgrade_started_at" } }],"must_not": [{ "exists": { "field": "upgraded_at" } }]}},"script": {"source": "ctx._source.upgraded_at = ctx._source.upgrade_started_at; ctx._source.upgrade_started_at = null;","lang": "painless"}}'
Finally, clean up the created service token:
curl -XDELETE --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"
Note that this procedure will apply with a fleet servers as well, as they are a special case of agent.