Fleet Server status stuck at updating, but seems to be working normally?!

Hi Martin,

Unfortunately, after upgrading, some agents can get stuck in "updating" state in Kibana, even if the actual underlying state is "healthy".

To un-mark any Elastic Agent as upgrading, you can follow the below commands as superuser.

Create a new service token:

curl -X POST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H' content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"

Next, do an update_by_query . The query finds all Elastic Agents which are marked updating and marks the update as complete. With the created token replace the ${TOKEN} below with that created token so the query can complete successfully:

curl -XPOST -H' Authorization: Bearer ${TOKEN}' -H' x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "upgrade_started_at" } }],"must_not": [{ "exists": { "field": "upgraded_at" } }]}},"script": {"source": "ctx._source.upgraded_at = ctx._source.upgrade_started_at; ctx._source.upgrade_started_at = null;","lang": "painless"}}'

Finally, clean up the created service token:

curl -XDELETE --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"

Note that this procedure will apply with a fleet servers as well, as they are a special case of agent.

1 Like