Enrolling Elastic Agent shows up in Fleet Agents but goes from "Updating" to "Offline"

Elastic Stack Elasticsearch
,
1

Hello,

I am trying to enroll Elastic Agent to different Windows Servers. I am running ELK and Fleet-Server via Docker. I am using the given commands by Kibana to install Elastic Agent and it says "Successfully enroled the Elastic Agent." The Agent shows up under "Fleet Agents" but then goes from "Updating" to "Offline". I don't know how or where to look what is going wrong...

Maybe somebody can help me!

Kind regards
Oliver

2

Hi Oliver, welcome to the community!

In order to investigate, we have to check the fleet-server logs and agent diagnostics to see if there are any errors. From the symptoms that the agent never goes to Healthy, there seems something wrong with the enrollment.

3

This is what the fleet-server-log says:

Elastic Agent successfully enrolled | log.level=info @timestamp=2023-12-07T10:18:13.644Z component={"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"} log={"source":"fleet-server-default"} ecs.version=1.6.0 service.name=fleet-server event.duration=540159862 http.request.id=01HH1WHXQCAY85NB8E1BMAS683 mod=enroll fleet.access.apikey.id=cLjIQ4wBDhRRmoi29_ft server.address= fleet.agent.id=c9c43654-d4e6-4871-af6f-c006f798e0d2 fleet.enroll.apikey.id=rPCpPowBvVJYkqdV31Qh fleet.policy.id=decf4020-9421-11ee-aeef-d51c0661d7e8 http.response.body.bytes=1874

ApiKey fail authentication | log.level=info @timestamp=2023-12-07T10:18:38.799Z component={"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"} log={"source":"fleet-server-default"} http.request.id=01HH1WJPRTF4XH5K1YHGGWJEB5 server.address= fleet.apikey.id=Fh8dP4wBRaQ9-NH2Omcu service.name=fleet-server event.duration=35355116 ecs.version=1.6.0 error.message=apikey auth response Fh8dP4wBRaQ9-NH2Omcu: [401 Unauthorized] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}

HTTP request error | log.level=info @timestamp=2023-12-07T10:18:38.799Z component={"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"} log={"source":"fleet-server-default"} ecs.version=1.6.0 http.request.id=01HH1WJPRTF4XH5K1YHGGWJEB5 server.address= http.response.status_code=400 service.name=fleet-server error.message=apikey auth response Fh8dP4wBRaQ9-NH2Omcu: [401 Unauthorized] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401} event.duration=35432654

And this is what happens on Windows:

PS C:\temp\elastic-agent-8.8.2-windows-x86_64> .\elastic-agent.exe enroll --url=https://<fleet-server>:8220 --enrollment-token=clBDcFBvd0J2VkpZa3FkVjMxUWg6TzdqOURabWtSVldfYUdxY0dPcktHQQ==
This will replace your current settings. Do you want to continue? [Y/n]:
{"log.level":"info","@timestamp":"2023-12-07T11:18:12.766+0100","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: https://<fleet-server>:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-12-07T11:18:13.908+0100","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":274},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
PS C:\temp\elastic-agent-8.8.2-windows-x86_64> & 'C:\Program Files\Elastic\Agent\elastic-agent.exe' status
State: STARTING
Message: Waiting for initial configuration and composable variables
Fleet State: FAILED
Fleet Message: status code: 400, fleet-server returned an error: BadRequest, message: apikey auth response Fh8dP4wBRaQ9-NH2Omcu: [401 Unauthorized] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate with provided credentials and anonymous access is not allowed for this request","additional_unsuccessful_credentials":"API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","ApiKey"]}},"status":401}
Components: (none)
4

The fleet server url doesn't seem correct (shouldn't contain <> characters), can you substitute the actual fleet server url?

5

This is just a placeholder to hide the real hostname.

6

Oh okay. It seems that the api key is not valid, can you try to create a new one on Fleet UI / Enrollment tokens tab and use that in the agent enroll command?
API key: api key [Fh8dP4wBRaQ9-NH2Omcu] has been invalidated

7

Tried that, generated a new enrollment token but still the same error messages.

The problem seems to have to do with Windows and enrolling on a previously installed agent , because a test-linux-machine could be enrolled without problems and is shown as healthy. When I uninstall agent on WIndows and reinstallt, it says the following:

State: HEALTHY
Message: Running
Fleet State: STOPPED
Fleet Message: Not enrolled into Fleet
Components: (none)
8

@Julia_Bardi do you have any idea where I could look at now?

EDIT: Nevermind, restarting the service in services.msc does not helpt but restarting it via Powershell helps (& 'C:\Program Files\Elastic\Agent\elastic-agent.exe' restart)

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.