Elastic agent is unhealthy

[lusynda]

Hi all,
I have a case when after i enroll the fleet server to Elasticsearch.
It become health for a while then became unhealthy.
I checked the log then it said this:

{"@timestamp":"2022-12-07T08:58:09.831566388Z","agent":{"id":"66a58979-0d90-9963-a3da-490d72b11ef0","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"error","origin":{"file":{"line":122,"name":"Http.cpp"}}},"message":"Http.cpp:122 HTTP code 401: Unauthorized","process":{"pid":830,"thread":{"id":953}}}
{"@timestamp":"2022-12-07T08:58:09.831638647Z","agent":{"id":"66a58979-0d90-9963-a3da-490d72b11ef0","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"error","origin":{"file":{"line":246,"name":"Client.cpp"}}},"message":"Client.cpp:246 HTTP Status Code (401): {\"error\":{\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id xxx\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]},\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"root_cause\":[{\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id xxx\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]},\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"type\":\"security_exception\"}],\"type\":\"security_exception\"},\"status\":401}","process":{"pid":830,"thread":{"id":953}}}
{"@timestamp":"2022-12-07T08:58:09.831671723Z","agent":{"id":"66a58979-0d90-9963-a3da-490d72b11ef0","type":"endpoint"},"ecs":{"version":"1.5.0"},"log":{"level":"notice","origin":{"file":{"line":84,"name":"BulkQueueConsumer.cpp"}}},"message":"BulkQueueConsumer.cpp:84 Elasticsearch connection is down","process":{"pid":830,"thread":{"id":953}}}

Elastic version i'm using is 8.5.2
For the api key i have no ideal where the agent got that authen key

Please help
Thank for your time.

[ferullo (Daniel Ferullo)]

Hi @lusynda . It looks like Endpoint cannot write to Elasticsearch because it's API key is invalid. Is the Agent on that host successfully connecting to the Stack? An easy way to know is if it appears active in Fleet. If it does, then assigning the Agent to a new policy (even one with the same settings) is an effective way to force an API key update, which should resolve your issue. In my testing, after reassigning the Agent policy I saw Agent go UNHEALTHY for a few minutes after switching the Agent policy but it did stabilize as HEALTHY. I hope that helps.

[lusynda]

Ok i've tried your solusion and it seems to be working.
after change the pocily and also change the config output elastic in the fleet settings the host seems to be healthy again.
But it seems like everytime i change the policy it takes a few minutes for the server to become healthy again.
Still it works so Thanks you