--insecure flag in fleet/elastic agent deployment to solve X509: certificate signed by unknown authority

hello everyone, hope you're doing well

i really love the elk stack and its community, much love to you guys !!

i have a question :
while deploying fleet or elastic agents i get this error message x509 to solved it I followed this Troubleshoot common problems | Fleet and Elastic Agent Guide [8.14] | Elastic thanks to @stephenb basically adding --insecure flag after deployment token

my question is is this just a bug that needs to be fixed and it's safe to use this approach to solve the issue or might it cause a security vulnerability ?

@Abdarrahmane

First --insecure is not a bug; this flag is provided to support quick start, this is simple underlying certs functionality validating certs / CA.

Once you have all the certs setup correctly you should be able to remove the --insecure flag did you try that... are you still getting the error?

A key to this is

if you don’t specify certificates when you set up Fleet Server, self-signed certificates are generated automatically.

Also from the docs here

If you set up all the certs and provided them to the Fleet server then it will use the certs and CA you provided ...

So the Meta Steps Are

Create All the Certs Before you install Fleet Server...
Install Fleet Server with the Certs you Provided and CA
Then When you install an Agent you have to provide the CA see here

If you are enrolling the agent in a Fleet Server that uses your organization’s certificate you must add the --certificate-authorities option to the command provided in the in-product instructions. If you do not include the certificate, you will see the following error: "x509: certificate signed by unknown authority".

Then you can use the CA when you install an agent and you should not need to use

If you do this all correctly, you should not need to use the `--insecure``

i did create all the certs using certsutil in elasticsearch and that before installing fleet server ofc
then installed fleet with --insecure flag
I tried removing the flag for agents installation in a windows and ubuntu endpoits but it didn't work

so should I mention the certifications in the elastic-agents.yml or elastic-agent.reference.yml

here are some important parts

# Fleet configuration
######################################
outputs:
  default:
    type: elasticsearch
    hosts: [127.0.0.1:9200]
    api_key: "example-key"
    #username: "elastic"
    #password: "changeme"
    preset: balanced
# fleet:
#   access_api_key: ""
#   kibana:
#     # kibana minimal configuration
#     hosts: ["localhost:5601"]
#     ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

#     # optional values
#     #protocol: "https"
#     #service_token: "example-token"
#     #path: ""
#     #ssl.verification_mode: full
#     #ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
#     #ssl.cipher_suites: []
#     #ssl.curve_types: []
#   reporting:
#     # Reporting threshold indicates how many events should be kept in-memory before reporting them to fleet.
#     #reporting_threshold: 10000
#     # Frequency used to check the queue of events to be sent out to fleet.
#     #reporting_check_frequency_sec: 30

my configuration
elasticsearch hosts: ["192.168.1.15:9200", "192.168.1.16:9200"]
under this cert elasticsearch-ca.pem

kibana 192.168.1.20:5601
with kibana-server.cert and kibana-server.key

and fleet server 192.168.1.23:8220 with fleet-server.crt and
fleet-server.key

But did you include the CAs on the agent install?

Also, I am confused. Are you trying to do fleet -managed or stand-alone... please pick one or the other...

Did you follow this example for the agent

Also, can you show the commands you used to install the fleet

And then show the exact command you used to install a different agent?

Most importantly, are you including you may need to put both the CAs there as a list

--certificate-authorities <string>
Comma-separated list of root certificates used for server verification.

i installed fleet server on a separate ubuntu server vm with the command

sudo ./elastic-agent install --url=https://192.168.1.23:8220
--fleet-server-es=https://192.168.1.15:9200
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjA4OTk4NzU0NTg6RUdHcWJ3U0xUOFdBUlZiemRCQ3ZZQQ --insecure
--fleet-server-policy=fleet-server-policy
--certificate-authorities=/etc/elastic_certs/elasticsearch-ca.pem
--fleet-server-es-ca=/etc/elastic_certs/elasticsearch-ca.pem
--fleet-server-cert=/etc/elastic_certs/fleet-server01.crt
--fleet-server-cert-key=/etc/elastic_certs/fleet-server01.key
--fleet-server-port=8220

and then deployed an elastic agent managed by fleet on an endpoint in this case one ubuntu desktop vm with the command
curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.14.3-linux-x86_64.tar.gz
tar xzvf elastic-agent-8.14.3-linux-x86_64.tar.gz
cd elastic-agent-8.14.3-linux-x86_64
sudo ./elastic-agent install --url=https://192.168.1.23:8220 --enrollment-token=TXVfQnJaQUJFekdremM2S2Y2SXQ6QmpHcEdIV2VUdjZVWEQ5WERHN0o0QQ== --insecure

is this correct ?

Here is my certs directory... I reused a few certs

Also key...

root@stephenb-es-8-test:/etc/elasticsearch# ls -lRrt certs/
certs/:
total 20
drwxr-xr-x 2 root root 4096 Jul 16 04:11 ca
-rw------- 1 root root 2517 Jul 16 04:11 elastic-stack-ca.zip
-rw-rw-rw- 1 root root 3596 Jul 16 04:15 elastic-certificates.p12
drwxr-xr-x 2 root root 4096 Jul 16 04:23 http_cert
-rw------- 1 root root 2621 Jul 16 04:23 certificate-bundle.zip

certs/ca:
total 8
-rw-rw-rw- 1 root root 1675 Jul 16 04:11 ca.key
-rw-rw-rw- 1 root root 1200 Jul 16 04:11 ca.crt

certs/http_cert:
total 8
-rw-rw-rw- 1 root root 1679 Jul 16 04:23 http_cert.key
-rw-rw-rw- 1 root root 1200 Jul 16 04:23 http_cert.crt

IMPORTANT: (as @leandrojmp and I found out another topic)

I also find the fingerprint does not always work the way we think so in the Fleet Elasticsearch output I put the whole CA as shown in the docs. I highly Recommend this for Self Signed...

And here is the command I used to enroll the fleet on the same server and you can see I did not use --insecure and it installed fine..
Yes this should be easier.

sbrown@stephenb-es-8-test:~/elastic-agent-8.14.3-linux-x86_64$ sudo ./elastic-agent install \ 
>   --url=https://10.168.0.12:8220 \
>   --fleet-server-es=https://10.168.0.12:9200 \
>   --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjExMDg5MTkzNTU6UWxHeFRfREJTZWV2QlJJbmRWZzcyUQ \
>   --fleet-server-policy=4c1c90af-9cdf-47f7-8f29-1428c0ca1853 \
>    --fleet-server-es-ca=/etc/elasticsearch/certs/ca/ca.crt \
>    --certificate-authorities=/etc/elasticsearch/certs/ca/ca.crt \
>    --fleet-server-cert=/etc/elasticsearch/certs/http_cert/http_cert.crt \
>    --fleet-server-cert-key=/etc/elasticsearch/certs/http_cert/http_cert.key \
>    --fleet-server-port=8220 \
>    --fleet-server-es-cert=/etc/elasticsearch/certs/http_cert/http_cert.crt \
>    --fleet-server-es-cert-key=/etc/elasticsearch/certs/http_cert/http_cert.key
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[=== ] Service Started  [7s] Elastic Agent successfully installed, starting enrollment.
[    ] Waiting For Enroll...  [10s] {"log.level":"info","@timestamp":"2024-07-16T05:49:40.228Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
[====] Waiting For Enroll...  [12s] {"log.level":"info","@timestamp":"2024-07-16T05:49:42.233Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":865},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
[====] Waiting For Enroll...  [16s] {"log.level":"info","@timestamp":"2024-07-16T05:49:46.236Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":846},"message":"Fleet Server - Running on policy with Fleet Server integration: 4c1c90af-9cdf-47f7-8f29-1428c0ca1853; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
[    ] Waiting For Enroll...  [16s] {"log.level":"info","@timestamp":"2024-07-16T05:49:46.692Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":517},"message":"Starting enrollment to URL: https://10.168.0.12:8220/","ecs.version":"1.6.0"}
[ ===] Waiting For Enroll...  [17s] {"log.level":"info","@timestamp":"2024-07-16T05:49:47.799Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-16T05:49:47.801Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":298},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[====] Done  [18s]                              
Elastic Agent has been successfully installed.
sbrown@stephenb-es-8-test:~/elastic-agent-8.14.3-linux-x86_64$ 

Then to enroll an agent from another host I had to copy the ca.crt and reference it... (it seems like I should not since I put the CA in the Elasticsearch Fleet output but it seems I needed both...

Yup it is a lot... that is all with self signed certs...

sbrown@stephenb-inference-test:~/elastic-agent-8.14.3-linux-x86_64$ sudo ./elastic-agent install --url=https://10.168.0.12:8220 --enrollment-token=Q0lNbHVwQUJ2cUlnMmRpNTJCc2Q6emNLaVEybTVRRXV4LVJNQ3MtQlZjQQ== --certificate-authorities=/tmp/ca.crt 
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:Y
[  ==] Service Started  [7s] Elastic Agent successfully installed, starting enrollment.
[====] Waiting For Enroll...  [7s] {"log.level":"info","@timestamp":"2024-07-16T06:36:34.506Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":517},"message":"Starting enrollment to URL: https://10.168.0.12:8220/","ecs.version":"1.6.0"}
[=   ] Waiting For Enroll...  [8s] {"log.level":"info","@timestamp":"2024-07-16T06:36:35.567Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":480},"message":"Restarting agent daemon, attempt 0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-07-16T06:36:35.569Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":298},"message":"Successfully triggered restart on running Elastic Agent.","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
[=   ] Done  [8s]                               
Elastic Agent has been successfully installed.

And then the logs and metrics show up...

Yes painful but there you go...

BUT if you use publicly signed certs from a recognized certificate authority... or use CSR and the CAs are installed in your OS most this goes away...

And finally the --insecure flag just .means certificate validation is not performed.. all comms are still via TLS..so you can make your choice

hello stephen!

it worked!
you're such a legend!

here is what I did:
firstly I noticed that you are using 'elastic-agent-8.14.3'
and I was using 'elastic-agent-8.14.1'
so I did the upgrade and then installed it via this command:

sudo ./elastic-agent install --url=https://192.168.1.23:8220 \
  --fleet-server-es=https://192.168.1.15:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjExMTc2OTY5OTk6eGx0U3hBNUNRMVNERU9XNFlsTWRVQQ \
  --fleet-server-policy=fleet-server-policy \
  --certificate-authorities=/etc/elastic_certs/elasticsearch-ca.pem \
  --fleet-server-es-ca=/etc/elasticsearch-ca.pem \
  --fleet-server-cert=/etc/elastic_certs/fleet-server01.crt \
  --fleet-server-cert-key=/etc/elastic_certs/fleet-server01.key \
  --fleet-server-port=8220 \
  --fleet-server-es-cert=/etc/elastic_certs/fleet-server01.crt \
  --fleet-server-es-cert-key=/etc/elastic_certs/fleet-server01.key

but then I got this error: Elastic Agent enrollment fails on the host with x509: cannot validate certificate for x.x.x.x because it doesn't contain any IP SANs
I red this Troubleshoot common problems | Fleet and Elastic Agent Guide [8.14] | Elastic
and this Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.14] | Elastic
And I remembered that my CA 'elastic-stack-ca.p12' is in p12 so I converted it, using OpenSSL:

openssl pkcs12 -in elastic-stack-ca.p12 -out cert.crt -clcerts -nokeys
openssl pkcs12 -in elastic-stack-ca.p12 -out private.key -nocerts -nodes

to get cert.crt and its private key private.key

then I went to generate the SSL certificate for fleet server using this command:

./bin/elasticsearch-certutil cert \
  --name fleet-server1 \
  --ca-cert cert.crt \
  --ca-key private.key \
  --dns fleet-server.HOMELAB.LAN \
  --ip 192.168.1.23 \
  --pem

so I got fleet-server1.crt and fleet-server1.key that I copied to my fleet server and installed my fleet server with success using this command

sudo ./elastic-agent install --url=https://192.168.1.23:8220 \
  --fleet-server-es=https://192.168.1.15:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE3MjExMTc2OTY5OTk6eGx0U3hBNUNRMVNERU9XNFlsTWRVQQ \
  --fleet-server-policy=fleet-server-policy \
  --certificate-authorities=/etc/elastic_certs/cert.crt \
  --fleet-server-es-ca=/etc/elastic_certs/cert.crt \
  --fleet-server-cert=/etc/elastic_certs/fleet-server1.crt \
  --fleet-server-cert-key=/etc/elastic_certs/fleet-server1.key \
  --fleet-server-port=8220 \
  --fleet-server-es-cert=/etc/elastic_certs/fleet-server1.crt \
  --fleet-server-es-cert-key=/etc/elastic_certs/fleet-server1.key

and finally installed the agent on endpoints where I copied my cert.crt

using this command:

./elastic-agent install --url=https://192.168.1.23:8220 --enrollment-token=OGZQYnVwQUJFekdremM2S3N1Rzg6Ym5zc2VCVENTOUtjLTJjRG9ITXY3dw== --certificate-authorities=/home/ubuntu/Desktop/cert.crt 

and that solved it!

Thank you so much Stephen!

1 Like

@Abdarrahmane So you did not add the cert in the elasticsearch output in the fleet setting like I showed...
Perhaps I went overboard LOL :slight_smile:
This is Good thanks for the interaction!

1 Like

yes i did add the cert as you mentioned in the the elasticsearch output in the fleet setting I just forgot to mention that

ssl:
  certificate_authorities:
  - |
    -----BEGIN CERTIFICATE-----
    MIIDSjCCAjKgAwIBAgIVAKlphSqJclcni3P83gVsirxzuDuwMA0GCSqGSIb3DQEB
    CwUAMDQxMjAwBgNVBAMTKUVsYXN0aWMgQ2VydGlmaWNhdGUgVG9vbCBBdXRvZ2Vu
    ZXJhdGVkIENBMB4XDTIxMDYxNzAxMzIyOVoXDTI0MDYxNjAxMzIyOVowNDEyMDAG
    A1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5lcmF0ZWQgQ0Ew
    ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDOFgtVri7Msy2iR33nLrVO
    /M/6IyF72kFXup1E67TzetI22avOxNlq+HZTpZoWGV1I4RgxiQeN12FLuxxhd9nm
    rxfZEqpuIjvo6fvU9ifC03WjXg1opgdEb6JqH93RHKw0PYimxhQfFcwrKxFseHUx
    DeUNQgHkMQhDZgIfNgr9H/1X6qSU4h4LemyobKY3HDKY6pGsuBzsF4iOCtIitE9p
    sagiWR21l1gW/lNaEW2ICKhJXbaqbE/pis45/yyPI4Q1Jd1VqZv744ejnZJnpAx9
    mYSE5RqssMeV6Wlmu1xWljOPeerOVIKUfHY38y8GZwk7TNYAMajratG2dj+v9eAV
    AgMBAAGjUzBRMB0GA1UdDgQWBBSCNCjkb66eVsIaa+AouwUsxU4b6zAfBgNVHSME
    GDAWgBSCNCjkb66eVsIaa+AouwUsxU4b6zAPBgNVHRMBAf8EBTADAQH/MA0GCSqG
    SIb3DQEBCwUAA4IBAQBVSbRObxPwYFk0nqF+THQDG/JfpAP/R6g+tagFIBkATLTu
    zeZ6oJggWNSfgcBviTpXc6i1AT3V3iqzq9KZ5rfm9ckeJmjBd9gAcyqaeF/YpWEb
    ZAtbxfgPLI3jK+Sn8S9fI/4djEUl6F/kARpq5ljYHt9BKlBDyL2sHymQcrDC3pTZ
    hEOM4cDbyKHgt/rjcNhPRn/q8g3dDhBdzjlNzaCNH/kmqWpot9AwmhhfPTcf1VRc
    gxdg0CTQvQvuceEvIYYYVGh/cIsIhV2AyiNBzV5jJw5ztQoVyWvdqn3B1YpMP8oK
    +nadUcactH4gbsX+oXRULNC7Cdd9bp2G7sQc+aZm
    -----END CERTIFICATE-----

big thanks again
have a good day !

1 Like