Fleet-server: http: server gave HTTP response to HTTPS client

Elastic Stack Kibana
1

Morning Elastic,

I was wondering could anyone help me with this issue on enrolling my Linux Agent into Fleet. I am getting this error message when I enroll my agent.

2022-03-31T08:41:12.570-0500    WARN    [tls]   tlscommon/tls_config.go:101     SSL/TLS verifications disabled.
2022-03-31T08:41:13.450-0500    INFO    cmd/enroll_cmd.go:454   Starting enrollment to URL: https://172.16.100.6:8220/
2022-03-31T08:41:13.552-0500    WARN    [tls]   tlscommon/tls_config.go:101     SSL/TLS verifications disabled.
Error: fail to enroll: fail to execute request to fleet-server: http: server gave HTTP response to HTTPS client
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html
Failed to start elastic-agent.service: Unit elastic-agent.service not found.
Failed to enable unit: Unit file elastic-agent.service does not exist.
2

Hello @secopsgeek I will need a bit more details about your setup. What command did you use to bootstrap your Elastic Agent that runs Fleet Server and also what command did you use to install your Elastic Agent that returns that log message?

3

@secopsgeek Are you running on-premise?

4

I am using my Elastic in a single deployment on Ubuntu on prem.
I used a script to automate the install like I always did but know since 7.17 I am no longer able to enroll via this process. Or by manual process as well from Kibana dashboard script.

username='elastic'
certIP='172.16.100.6'
fleetIP='172.16.100.6'
token='Y2hzdzRIOEJBSU80Mmo3R2hKdzM6V1B2UWxOTHdTRENmTFFrZ013Rk5xdw=='

## Download the cert and add the the shared certs directory
# scp <username>@<Server IP>:<filename i.e. ca.crt> .
# mv ca.crt /usr/local/share/ca-certificates/
# OR #
scp $username@$certIP:/ca.crt /usr/local/share/ca-certificates/

## Add the cert
update-ca-certificates

## Download and install the tarball image
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-7.17.1-linux-x86_64.tar.gz
tar xzvf elastic-agent-7.17.1-linux-x86_64.tar.gz

## Since the agent was installed via apt we just need to enroll with the Fleet Server
cd elastic-agent-7.17.1-linux-x86_64
./elastic-agent install -f --url=https://$fleetIP:8220 --insecure --enrollment-token=$token

## Restart and enable the service
systemctl start elastic-agent.service
systemctl enable elastic-agent.service

cd ..
rm -rf elastic*
5

How did you setup the Elastic Agent on $fleetIP? What argument did you use? I see that you are using --insecure on this part and want to see the other side.

6

Because I am using a self signed certificate on my server.
I am using these security settings on my server:

image

7

I've looked at the instructions and they look ok to me.

In the fleet server logs do you have this Exposed over insecure HTTP; enablement of TLS is strongly recommended?

8

No sure, is there a guide I can check to see if this is needed to be set?

9

I know it's handling HTTP connection instead of HTTPS, I just don't know how to set it for HTTPS.

10

No worries, I am also trying to figure that out, Let's work together on this. I am looking at the code to understand how you can get into that state.

Did you upgrade from a previous version?

11

I did once before from 7.16.x then upgrade to 7.17 but this build is from ground up on 7.17.x

12

@secopsgeek It seems that you are running Fleet Server in insecure mode. When you installed/enrolled Fleet Server, did you have the --fleet-server-insecure-http option set? If so, you'll need to use an http protocol for your Fleet Server host. If you uninstall that agent and remove the flag, you'll be able to use https.

@pierhugues It seems we provide this option in the Fleet Server command when quick start is enabled, but I'm not sure why. I think we should not add --fleet-server-insecure-http to allow Fleet Server to generate self-signed certs. It worked for me locally when I tried this (on 8.2.0).

image
image1230×972 121 KB

13

Ok thanks, let me try that now.

14

image
image1345×282 104 KB

15

It worked:

elastic-agent-7.17.1-linux-x86_64/elastic-agent
2022-03-31T12:05:20.239-0500    WARN    [tls]   tlscommon/tls_config.go:101     SSL/TLS verifications disabled.
2022-03-31T12:05:20.605-0500    INFO    cmd/enroll_cmd.go:454   Starting enrollment to URL: https://172.16.100.6:8220/
2022-03-31T12:05:20.707-0500    WARN    [tls]   tlscommon/tls_config.go:101     SSL/TLS verifications disabled.
2022-03-31T12:05:21.469-0500    INFO    cmd/enroll_cmd.go:254   Successfully triggered restart on running Elastic Agent.
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.
16

My windows agent is installed:

PS C:\> C:\Users\ronniewatson\Desktop\test.ps1
WARNING: The version '3.0.1' of module 'Posh-SSH' is currently in use. Retry the operation after closing the applications.
VERBOSE: Performing the operation "Import certificate" on target "Item: C:\Users\ronniewatson\ca.crt Destination: Root".


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Root

Thumbprint                                Subject                                                                                                    
----------                                -------                                                                                                    
33C808A370A742271A5B4D1A7F755C09BBC69322  CN=Elastic Certificate Tool Autogenerated CA                                                               
33C808A370A742271A5B4D1A7F755C09BBC69322  CN=Elastic Certificate Tool Autogenerated CA                                                               
./elastic-agent : 2022-03-31T12:12:22.387-0500	WARN	[tls]	tlscommon/tls_config.go:101	SSL/TLS verifications disabled.
At C:\Users\ronniewatson\Desktop\test.ps1:45 char:1
+ ./elastic-agent install -f --url=https://172.16.100.6:8220 --insecure ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022-03-31T12:1...tions disabled.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
2022-03-31T12:12:23.348-0500	INFO	cmd/enroll_cmd.go:454	Starting enrollment to URL: https://172.16.100.6:8220/
2022-03-31T12:12:23.726-0500	WARN	[tls]	tlscommon/tls_config.go:101	SSL/TLS verifications disabled.
2022-03-31T12:12:27.273-0500	INFO	cmd/enroll_cmd.go:254	Successfully triggered restart on running Elastic Agent.
Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

Thanks, wow can't believe that it was the --insecure-HTTP that was the issue. I am still new to Fleet server and this was fun to learn.

Made me better at troubleshooting this by learning this fix.

17

The Windows agent enrolled but no Endpoint agents were coming up so I remove the --insecure-HTTP from my script and this cam up. I tried putting in the CA Cert in the policy under TLS setting for Endpoint agents like James showed me and this is what I got.

PS C:\Users\ronniewatson> C:\Users\ronniewatson\Desktop\test.ps1
WARNING: The version '3.0.1' of module 'Posh-SSH' is currently in use. Retry the operation after closing the applications.
VERBOSE: Performing the operation "Import certificate" on target "Item: C:\Users\ronniewatson\ca.crt Destination: Root".


   PSParentPath: Microsoft.PowerShell.Security\Certificate::LocalMachine\Root

Thumbprint                                Subject                                                                                                    
----------                                -------                                                                                                    
33C808A370A742271A5B4D1A7F755C09BBC69322  CN=Elastic Certificate Tool Autogenerated CA                                                               
33C808A370A742271A5B4D1A7F755C09BBC69322  CN=Elastic Certificate Tool Autogenerated CA                                                               
./elastic-agent : 2022-03-31T12:22:23.386-0500	INFO	cmd/enroll_cmd.go:454	Starting enrollment to URL: https://172.16.100.6:8220/
At C:\Users\ronniewatson\Desktop\test.ps1:45 char:1
+ ./elastic-agent install -f --url=https://172.16.100.6:8220 --enrollme ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (2022-03-31T12:2....16.100.6:8220/:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html

So the --insecure-HTTp works to get fleet agent installed but not with Agents or even with the command you told me above.

18

This shows you were able to enroll elastic-agent on windows, but endpoint didn't successfully start, Can you look at the logs if there any errors? you can get the logs by using ./elastic-agent diagnostics collect -f mylog.zip and opening the zip. I am interested in any error messages.

19 
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1270784,"@timestamp":"2022-03-31T18:06:24.386Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1593835,"@timestamp":"2022-03-31T18:06:24.387Z","message":"fail checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","http.request.id":"","mod":"enroll","fleet.enroll.apikey.id":"6NXQA38BZBchmXn3N8vw","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","fleet.policy.id":"08e561e0-8f57-11ec-89d4-6be0e336dfeb","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","http.response.body.bytes":1820,"event.duration":548217507,"@timestamp":"2022-03-31T18:11:58.494Z","message":"Elastic Agent successfully enrolled"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","req.LocalMeta":{"elastic":{"agent":{"id":"61c95d94-8276-4a12-b368-22f613d5e0a7","version":"7.17.1","snapshot":false,"build.original":"7.17.1 (build: 1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff at 2022-02-24 09:30:45 +0000 UTC)","upgradeable":true,"log_level":"info"}},"host":{"architecture":"x86_64","hostname":"WIS-DT01","name":"WIS-DT01","id":"0a7631d3-5a37-43d0-bd0f-bbbe74f13259","ip":["fe80::b12e:b176:bcad:1427/64","169.254.20.39/16","fe80::78fb:7e9c:5b5e:81f6/64","172.16.100.142/24","fe80::e086:5a7e:f729:e2fc/64","192.168.56.1/24","fe80::24f4:8ca0:13ca:7452/64","169.254.116.82/16","fd00::ffff:ac10:1/112","fe80::cc59:5a74:a50:2122/64","169.254.33.34/16","fe80::580c:8e1d:5594:a36d/64","192.168.217.1/24","fe80::a9db:c3c:af1e:2634/64","192.168.10.1/24","::1/128","127.0.0.1/8","fe80::44c:f68f:cfe4:41cf/64","172.25.48.1/20","fe80::213d:c10:3f79:ab90/64","172.29.16.1/20","fe80::b82e:3483:3000:3608/64","172.20.96.1/20","fe80::5004:a6cc:11be:a9b2/64","172.17.176.1/20","fe80::58dc:13bd:8776:bd/64","172.26.0.1/20"],"mac":["8c:16:45:bc:ef:0d","5c:ff:35:d4:3b:d7","0a:00:27:00:00:0c","00:ff:84:07:e5:03","00:ff:9e:cc:bc:1f","00:50:56:c0:00:01","00:50:56:c0:00:02","00:15:5d:51:c2:a9","00:15:5d:06:a5:3e","00:15:5d:d5:63:a2","00:15:5d:70:6b:e6","00:15:5d:52:13:10"]},"os":{"family":"windows","kernel":"10.0.22000.593 (WinBuild.160101.0800)","platform":"windows","version":"10.0","name":"Windows 10 Pro","full":"Windows 10 Pro(10.0)"}},"@timestamp":"2022-03-31T18:12:03.314Z","message":"applying new local metadata"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ctx":"processPolicy","policyRevision":15,"policyCoordinator":1,"fleet.policy.id":"08e561e0-8f57-11ec-89d4-6be0e336dfeb","hash.sha256":"15480a20978f6110603014b49101d4170f23170e888aef52de4a5a36041aa276","fleet.default.apikey.id":"bCUt4X8BAIO42j7GaOGI","@timestamp":"2022-03-31T18:12:03.851Z","message":"Updating agent record to pick up default output key."}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:12:04.105Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:14:16.946Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1195164,"@timestamp":"2022-03-31T18:15:29.114Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1251324,"@timestamp":"2022-03-31T18:15:29.114Z","message":"fail checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:16:07.08Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:17:57.027Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:19:46.616Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"61c95d94-8276-4a12-b368-22f613d5e0a7","http.request.id":"","fleet.access.apikey.id":"riUt4X8BAIO42j7GUuCU","ackToken":"","createdAt":"2022-03-31T17:22:12.061Z","id":"policy:08e561e0-8f57-11ec-89d4-6be0e336dfeb:15:1","type":"POLICY_CHANGE","inputType":"","timeout":0,"@timestamp":"2022-03-31T18:21:36.851Z","message":"Action delivered to agent on checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1152625,"@timestamp":"2022-03-31T18:22:03.874Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1199605,"@timestamp":"2022-03-31T18:22:03.874Z","message":"fail checkin"}
{"log.level":"error","service.name":"fleet-server","service.name":"fleet-server","message":"http: TLS handshake error from 172.16.100.142:6161: remote error: tls: bad certificate\n","@timestamp":"2022-03-31T18:23:33.212Z"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1162856,"@timestamp":"2022-03-31T18:27:21.499Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1217495,"@timestamp":"2022-03-31T18:27:21.499Z","message":"fail checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1150092,"@timestamp":"2022-03-31T18:36:11.793Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1216405,"@timestamp":"2022-03-31T18:36:11.793Z","message":"fail checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1294986,"@timestamp":"2022-03-31T18:43:13.203Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1385743,"@timestamp":"2022-03-31T18:43:13.203Z","message":"fail checkin"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","fleet.apikey.id":"MaZjFn0B_K2qM-ySCs8c","http.request.id":"","event.duration":1068136,"@timestamp":"2022-03-31T18:53:08.207Z","message":"ApiKey fail authentication"}
{"log.level":"info","service.name":"fleet-server","service.name":"fleet-server","fleet.agent.id":"a5ae3698-cf0c-47f7-9977-086acf0450da","http.request.id":"","error.message":"apikey auth response MaZjFn0B_K2qM-ySCs8c: [401 Unauthorized] {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}}],\"type\":\"security_exception\",\"reason\":\"unable to authenticate with provided credentials and anonymous access is not allowed for this request\",\"additional_unsuccessful_credentials\":\"API key: unable to find apikey with id MaZjFn0B_K2qM-ySCs8c\",\"header\":{\"WWW-Authenticate\":[\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\",\"Bearer realm=\\\"security\\\"\",\"ApiKey\"]}},\"status\":401}","http.response.status_code":400,"event.duration":1124841,"@timestamp":"2022-03-31T18:53:08.207Z","message":"fail checkin"}
20

Are those logs from the Windows machine where Endpoint won't run or from Fleet Server? If they're from Fleet Server, can you check the logs from the Windows machine?

I have a few Endpoint related questions:

  1. What Windows version are you running?
  2. Are you using 7.17.1 on Windows too? (side note, 7.17.2 is out now and has Endpoint bug fixes in it)
  3. Are Endpoint files on disk? Is it running? Endpoint installs to c:\Program Files\Elastic\Endpoint and runs as elastic-endpoint.exe
  4. If there are logs on the Windows machine in c:\Program Files\Elastic\Endpoint\state\log can you look for logs from when Endpoint connects to Fleet Server and/or Elasticsearch? Searching for Http.cpp and checking the message is about a failing connection to one of those destinations is a good way to find them.