Fleet Server - Error - x509: certificate signed by unknown authority

[ethical20]

Hi All,

Please have some patience with me cause i ll give as much details.

I have my ELK stack (elastic, kibana, logstash) on same local server (192.168.0.2) and im trying to install Fleet server on another host (192.168.0.3) but i'm getting this error:

{"log.level":"info","@timestamp":"2022-08-25T09:36:24.534Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Elastic Certificate Tool Autogenerated CA\")","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html

This is my elasticsearch.yml

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/all-elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/all-elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: "/etc/elasticsearch/http.p12"

#-------------------------------Fleet------------------------------------
xpack.security.authc.api_key.enabled: true

I've followed this documentation to create the certs:

Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent...

Here the steps in details

openssl pkcs12 -in all-elastic-certificates.p12 -out elasticsearch-ca.crt -clcerts -nokeys
openssl pkcs12 -in all-elastic-certificates.p12 -out elasticsearch-ca.key -nocerts -nodes

./bin/elasticsearch-certutil ca --pem

I've got fleet-ca.crt and fleet-ca.key

The i've run this:

sudo ./bin/elasticsearch-certutil cert \
  --name fleet-server \
  --ca-cert fleet-ca/fleet-ca.crt \
  --ca-key fleet-ca/fleet-ca.key \
  --dns localhost,node-1 \
  --ip 0.0.0.0,192.168.0.2,192.168.0.3,127.0.0.1,::1 \
  --pem

This gave me fleet-server.crt and fleet-server.key

added the elasticsearch-ca.crt into the fleet settings

Finally This is the installation code I run on fleet server

sudo ./elastic-agent install \
--url=https://192.168.0.3:8220 \
  --fleet-server-es=https://192.168.0.2:9200 \
  --fleet-server-service-token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
  --fleet-server-policy=XXXXXXXXX-8d2d-cddeef5ffe8c \
  --certificate-authorities=/etc/certs/elasticsearch-ca.crt \
  --fleet-server-es-ca=/etc/certs/fleet-ca.crt \
  --fleet-server-cert=/etc/certs/fleet-server.crt \
  --fleet-server-cert-key=/etc/certs/fleet-server.key

Is there anything wrong from my side? Please help im stuck in this since 3 days!

[Kevin_Clukey]

Hi there.

--fleet-server-es-ca is the elasticsearch CA. So in your case it would be elasticsearch-ca.crt.

--certificate-authorities is a list of root certs for server verification. In your case this would probably be fleet-ca.crt

[ethical20]

@ethical20

fleet-serve

Hi @Kevin_Clukey

Thanks for the reply I've edited it as you advised but sadly same error:

sudo ./elastic-agent install \
> --url=https://192.168.0.3:8220 \
>   --fleet-server-es=https://192.168.0.2:9200 \
>   --fleet-server-service-token=XXXXXXXXXXXXXXXXXXXXXXXXX \
>   --fleet-server-policy=XXXXXXXXXXX-cddeef5ffe8c \
>   --certificate-authorities=/etc/certs/fleet-ca.crt \
>   --fleet-server-es-ca=/etc/certs/elasticsearch-ca.crt \
>   --fleet-server-cert=/etc/certs/fleet-server.crt \
>   --fleet-server-cert-key=/etc/certs/fleet-server.key
[sudo] password for fleet:
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-08-25T14:10:55.426Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Error - x509: certificate signed by unknown authority","ecs.version":"1.6.0"}

[Kevin_Clukey]

@ethical20

It looks like you may have grabbed the ca cert from the transport layer (although if you used the built in certs in 8.x I believe they are signed by the same built in CA).

If you used the built in certs for elasticsearch, use the http_ca.crt in /etc/elasticsearch/certs as the CA cert for --fleet-server-es-ca.

[Kevin_Clukey]

@ethical20
The better question is what CA did you used to sign your http.p12 certificate?

That is the ca.crt that is needed for --fleet-server-es-ca. If you followed the steps here then you would need to run to get the ca cert pem formatted.

openssl pkcs12 -in elastic-stack-ca.p12 -out elasticsearch-ca.crt -clcerts -nokeys

[ethical20]

Thanks @Kevin_Clukey

This solved the issue, actually I was generating the elasticsearch-ca.crt from a CA other than the one I signed the http.p12 , after i did what you advised it worked now and the enrollment was successful.

Successfully enrolled the Elastic Agent.
Elastic Agent has been successfully installed.

Actually the naming in the elastic documentation was confusing the -fleet-server-es-ca and the --certificate-authorities .

I have a new error now when trying to install elastic agent on a windows machine to test the connection with Fleet server. After downloading the agent and trying to run i got this error:

 .\elastic-agent.exe install --url=https://192.168.0.3:8220 --enrollment-token=XXXXXXXXXX
Elastic Agent will be installed at C:\Program Files\Elastic\Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2022-08-25T09:22:29.280-0700","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://192.168.0.3:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html

Any clues?

[Kevin_Clukey]

@ethical20

When enrolling an agent to fleet, you need to specify the CA that signed the fleet server certificate. This can be found in the documentation here a the bottom of the page.

sudo elastic-agent install -f --url=https://192.0.2.1:8220 \
  --enrollment-token=<string> \
  --certificate-authorities=/path/to/ca.crt

[ethical20]

@Kevin_Clukey

Many thanks for your help, now i have successfully created the fleet server and the agent is connected to it.

Both server and agent were green and healthy but after few minutes both turned yellow and unhealthy.

if i restart them they become healthy for few minutes and then turn to unhealthy.

In the minutes where both are healthy there are no logs coming.

I've checked the agent logs and found this

State changed to FAILED: 1 error occurred:\n\t* 1 error: open /usr/share/elasticsearch/elasticsearch-ca.crt: The system cannot find the path specified. reading <nil> accessing 'elasticsearch'\n\n - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}

Checking fleet settings:

I can see the path is correct and i checked it from the elastic-stack I can confirm the file is there and accessible.

Your help is really appreciated.

Thanks in advance.

[ethical20]

Actually I've solved this by pasting the certificate directly here... I don't know why for some reason the cert path is not readable!

Thanks