[elastic_agent][error] Cannot checkin in with fleet-server, retrying

Elastic Stack Elastic Agent
1

Hi everyone

All my agents are showing offline and I am not sure why.
Here are some logs and info to start the discussion..!

fleet
fleet1186×428 50.1 KB

fleet2
fleet2976×386 64.9 KB

{
"log.level": "info",
"@timestamp": "2024-01-23T06:21:39.097Z",
"message": "ApiKey fail authentication",
"component": {
"binary": "fleet-server",
"dataset": "elastic_agent.fleet_server",
"id": "fleet-server-default",
"type": "fleet-server"
},
"log": {
"source": "fleet-server-default"
},
"service.type": "fleet-server",
"server.address": "",
"ecs.version": "1.6.0",
"[service.name](http://service.name)": "fleet-server",
"[http.request.id](http://http.request.id)": "01HMTFN464ZQ2G3J9DJF37W6GT",
"error.message": "unauthorized: apikey auth response 5iaUM4wBn1bodLGK1xXo: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id 5iaUM4wBn1bodLGK1xXo\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id 5iaUM4wBn1bodLGK1xXo\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}",
"[fleet.apikey.id](http://fleet.apikey.id)": "5iaUM4wBn1bodLGK1xXo",
"event.duration": 2653200
}
2

Hi,

check if the API key was deleted from Elasticsearch or the API key is expired.

Regards