Elastic-agent-complete->x509: certificate signed by unknown authority

Hi all,

i am following this -> Monitor resources on private networks | Elastic Observability [8.16] | Elastic

sudo docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL=https://elastic:8220 \
  --env FLEET_ENROLLMENT_TOKEN=token \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --env FLEET_INSECURE=false \
  --env KIBANA_FLEET_CA='/usr/share/elastic-agent/certs/ca.crt' \
  --env KIBANA_FLEET_USERNAME='elastic' \
  --env KIBANA_FLEET_PASSWORD='password' \
  --env FLEET_SERVER_ELASTICSEARCH_CA='/usr/share/elastic-agent/certs/ca.crt' \
  -v /ca.crt:/usr/share/elastic-agent/certs/ca.crt \
  --add-host elastic:ip_address \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:8.16.4

getting error ->

{"log.level":"info","@timestamp":"2025-02-26T10:02:53.681Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":526},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://elastic:8220/ with exponential backoff (init 1s, max 10s)","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority

Thank you in advance.

Hi @Amol_Nagotkar,

This could be a certificate issue with Fleet. Have you tried following this guide to enable TLS?

Let us know if that helps!

Hi ma'am

By the way, i am doing all this to run synthetic monitoring. currently it is just stuck in Test result PENDING state, even for google.com

actually we are using ssl certificates.
same certificates i used for elastic and kibana, so ideally it should work for elastic-agent-complete too.
now

sudo docker run -d \
  --name elastic-agent \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL=https://elastic-uat.helo.ai:8220 \
  --env FLEET_ENROLLMENT_TOKEN=token \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --env FLEET_INSECURE=false \
  --env FLEET_CA=/usr/share/elastic-agent/certs/helo.ai.crt \
  --env KIBANA_FLEET_HOST=https://kibana-uat.helo.ai:5601 \
  --env KIBANA_FLEET_USERNAME=elastic \
  --env KIBANA_FLEET_PASSWORD=password \
  --env ELASTICSEARCH_HOSTS=https://10.40.13.132:9200 \
  --env ELASTICSEARCH_SSL_CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  --env ELASTICSEARCH_USERNAME=elastic \
  --env ELASTICSEARCH_PASSWORD=password \
  --env ELASTICSEARCH_SSL_VERIFICATION_MODE=certificate \
  -v ./helo.ai.crt:/usr/share/elastic-agent/certs/helo.ai.crt \
  -v ./helo.ai.key:/usr/share/elastic-agent/certs/helo.ai.key \
  -v ./helo.ai.crt:/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  --add-host elastic-uat.helo.ai:10.40.13.133 \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:8.16.3

I am getting error ->

{"log.level":"error","@timestamp":"2025-02-27T09:42:25.217Z","message":"Failed to connect to backoff(elasticsearch(https://10.40.13.132:9200)): Get \"https://10.40.13.132:9200\": x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-02-27T09:42:25.217Z","message":"Attempting to reconnect to backoff(elasticsearch(https://10.40.13.132:9200)) with 5 reconnect attempt(s)","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":139,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-02-27T09:42:25.229Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"server.address":"10.40.13.132:9200","ecs.version":"1.6.0","log.logger":"esclientleg","service.name":"metricbeat","network.transport":"tcp","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-02-27T09:42:27.246Z","message":"Failed to connect to backoff(elasticsearch(https://10.40.13.132:9200)): Get \"https://10.40.13.132:9200\": x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"service.name":"filebeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

i tried keeping FLEET_INSECURE=true also. still getting same issue.

Thank you in advance.

I am using ssl certificate still it is giving error for tls
our certificates are NOT self-signed certificate.
it is certificates issued by a custom Certificate Authority (CA).

version: '3.0'

services:
  elastic-agent:
    image: docker.elastic.co/elastic-agent/elastic-agent-complete:8.16.3
    container_name: elastic-agent
    environment:
      - FLEET_ENROLL=1
      - FLEET_URL=https://elastic-uat.helo.ai:8220
      - FLEET_ENROLLMENT_TOKEN=token
      - SSL_CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - LOG_LEVEL=debug
      - ELASTICSEARCH_HOSTS=https://elastic.helo.ai:9200
      - FLEET_CA=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - ELASTICSEARCH_SSL_CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/certs/ca/helo.ai.crt
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/certs/ca/helo.ai.key
      - FLEET_SERVER_ES_CA=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - ELASTICSEARCH_USERNAME=elastic
      - ELASTICSEARCH_PASSWORD=password
      - FLEET_INSECURE=true
      - fleet-server-es=https://elastic.helo.ai:9200
      - fleet-server-es-ca=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - certificate-authorities=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - fleet-server-cert=/usr/share/elastic-agent/certs/ca/helo.ai.crt
      - fleet-server-cert-key=/usr/share/elastic-agent/certs/ca/helo.ai.key
      - ELASTICSEARCH_SSL_VERIFICATION_MODE=certificate
      - FLEET_SERVER_ENABLE=true
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://elastic-uat.helo.ai:8220
      - FLEET_SERVER_SERVICE_TOKEN=token
    privileged: true  # Add this line to enable privileged mode
    cap_add:
      - NET_RAW
      - SETUID
    extra_hosts:
      - "elastic-uat.helo.ai:10.40.13.133"
      - "elastic.helo.ai:10.40.13.132"
    volumes:
      - /var/ssl/helo.ai/helo.ai.bundle.crt:/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt
      - /var/ssl/helo.ai/helo.ai.key:/usr/share/elastic-agent/certs/ca/helo.ai.key
      - /var/ssl/helo.ai/helo.ai.crt:/usr/share/elastic-agent/certs/ca/helo.ai.crt
    restart: unless-stopped

error---------

elastic-agent    | {"log.level":"error","@timestamp":"2025-02-27T15:09:58.543Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: tls: failed to verify certificate: x509: certificate signed by unknown authority","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}


even with the below command it is giving me same error ->

sudo docker run \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL=https://fleet.helo.ai:8220 \
  --env FLEET_ENROLLMENT_TOKEN=token \
  --env insecure=true \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  --add-host fleet.helo.ai:10.40.13.133 \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:8.16.4

error -------

agent container initialisation - effective capabilities
agent container initialisation - ambient capabilities
agent container initialisation - chown paths
{"log.level":"info","@timestamp":"2025-02-28T08:36:10.388Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":520},"message":"Starting enrollment to URL: https://fleet.helo.ai:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-02-28T08:36:10.668Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":526},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://fleet.helo.ai:8220/ with exponential backoff (init 1s, max 10s)","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.16/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.16/fleet-troubleshooting.html

@Amol_Nagotkar can you try following the steps for this error as listed in the troubleshooting documentation? There are steps to address in there.

hi ma'am

i read above doc. I am able to create elastic-agent successfully. but my synthetic monitor is not working.

  1. installed fleet-server
  2. installed elastic-agent
  3. created private location
  4. synthetic monitor get stuck in manually test pending state

as per this doc - Run Elastic Agent in a container | Fleet and Elastic Agent Guide [8.16] | Elastic

i have to install elastic-agent-complete first to run synthetic monitoring, for which i am getting error. so i uninstalled elastic-agent and installed elastic-agent-complete.

elastic-agent    | {"log.level":"error","@timestamp":"2025-02-27T15:09:58.543Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fleet-server-default (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: tls: failed to verify certificate: x509: certificate signed by unknown authority","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default","type":"output","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

my aim is to run synthetic monitoring for which i am doing above stuff. let me know if i am missing anything.

Thank you in advance.

So the issue is the certification configuration for Elastic Agent and Fleet, which Synthetic Monitoring uses. Until you fix the agent certificate issue the monitors will fail.

I did a quick scan of your configuration against the SSL/TLS guide and I can't see any obvious misconfiguration.

I assume you are using a self-managed fleet server? I would recommend checking out the configuring TLS guide and also see if you need to specify the certs in Kibana as outline in this similar issue.

Alternatively you could see if regenerating the certifications is an option.

Can you try either of those and let us know if that resolves the issue?

Hi @Amol_Nagotkar,

The error you're seeing implies that the agent cannot verify the connection to Elasticsearch using the provided CAs. You'll need to make sure that FLEET_SERVER_ELASTICSEARCH_CA is correctly pointing to the correct CA for Elasticsearch cert.

If that is correct, you can try setting the cert signature in the default output configuration. In Kibana -> Fleet -> Settings -> Outputs -> <default ES>, put the requested value in Elasticsearch CA trusted fingerprint field.

If it's still not working after that, you might want to request a diagnostics file and review the agent policy to make sure the right certificate values are set in the policy.

Hi sir,

thanks for the reply

after adding above flag. my command looks like this ->

  sudo docker run -d \
  --name elastic-agent \
  --add-host fleet.helo.ai:10.40.13.133 \
  --add-host elastic-uat.helo.ai:10.40.13.132 \
  --env FLEET_ENROLL=1 \
  --env FLEET_URL=https://fleet.helo.ai:8220 \
  --env FLEET_SERVER_ELASTICSEARCH_HOST=https://elastic-uat.helo.ai:9200 \
  --env FLEET_SERVER_INSECURE_HTTP=false \
  --env FLEET_ENROLLMENT_TOKEN=token \
  --env FLEET_SERVER_ELASTICSEARCH_CA=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  --env FLEET_SERVER_ES_CERT_KEY=/usr/share/elastic-agent/certs/ca/helo.ai.key \
  --env FLEET_SERVER_ES_CERT=/usr/share/elastic-agent/certs/ca/helo.ai.crt \
  --env ELASTICSEARCH_USERNAME=elastic \
  --env ELASTICSEARCH_PASSWORD=password \
  --env SSL_CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  --env ELASTICSEARCH_SSL_CERTIFICATE_AUTHORITIES=/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  --env ELASTIC_AGENT_CERT_KEY=/usr/share/elastic-agent/certs/ca/helo.ai.key \
  --env ELASTIC_AGENT_CERT=/usr/share/elastic-agent/certs/ca/helo.ai.crt \
  --env LOG_LEVEL=debug \
  --env FLEET_INSECURE=true \
  --env ELASTICSEARCH_SSL_VERIFICATION_MODE=certificate \
  --env user=elastic-agent \
  --cap-add=NET_RAW \
  --cap-add=SETUID \
  -v /var/ssl/helo.ai/helo.ai.bundle.crt:/usr/share/elastic-agent/certs/ca/helo.ai.bundle.crt \
  -v /var/ssl/helo.ai/helo.ai.crt:/usr/share/elastic-agent/certs/ca/helo.ai.crt \
  -v /var/ssl/helo.ai/helo.ai.key:/usr/share/elastic-agent/certs/ca/helo.ai.key \
  --rm docker.elastic.co/elastic-agent/elastic-agent-complete:8.16.3

error ->

{"log.level":"info","@timestamp":"2025-02-28T11:44:44.588Z","message":"'ca_trusted_fingerprint' set, looking for matching fingerprints","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"tls","log.origin":{"file.line":180,"file.name":"tlscommon/tls_config.go","function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2025-02-28T11:44:44.588Z","message":"no CA certificate matching the fingerprint","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"service.name":"metricbeat","ecs.version":"1.6.0","log.logger":"tls","log.origin":{"file.line":209,"file.name":"tlscommon/tls_config.go","function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.trustRootCA"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-02-28T11:44:44.589Z","message":"Error dialing x509: certificate signed by unknown authority","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"network.transport":"tcp","server.address":"elastic-uat.helo.ai:9200","ecs.version":"1.6.0","service.name":"metricbeat","log.logger":"esclientleg","log.origin":{"file.line":39,"file.name":"transport/logging.go","function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.LoggingDialer.func2"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-02-28T11:44:44.604Z","message":"Failed to connect to backoff(elasticsearch(https://elastic-uat.helo.ai:9200)): Get \"https://elastic-uat.helo.ai:9200\": x509: certificate signed by unknown authority","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"service.name":"filebeat","ecs.version":"1.6.0","log.logger":"publisher_pipeline_output","log.origin":{"file.line":148,"file.name":"pipeline/client_worker.go","function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run"},"ecs.version":"1.6.0"}

dont know how but same certificates i used in elasticsearch setup, there it is working perfectly.
plus as mentioned in doc -> Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.16] | Elastic

i added sha-256 hexcode of my CA here ->


plus

outputs is not showing any status here.

but no luck. :tired_face:

When you enroll in fleet, almost none of the other fleet settings in the current list (other than fleet URL, enroll, and the enrollment token) matter anymore.

Similarly, none of the Elasticsearch settings matter either as you're getting your configuration from fleet.

So you have two options:

  1. Add the public certificate of your certificate authority into the docker container and update the ca certificates (annoying) or
  2. Add the ca certs fingerprint to the output in fleet

You've decided to do number 2 but it is not working. When the trusted fingerprint setting is not working it is almost always caused by one of two things.

  1. You've incorrectly generated the sha256 of your certificate.

Or

  1. You've incorrectly setup the certificate on your Elasticsearch node and it does not include the full ca chain.

To check for number 2 simply open your Elasticsearch (at Port 9200, not Kibana!) cluster in a browser and check the certificate, is it all by itself or does it show a certificate chain? Does it go all the way up to your root CA?

If the certificate chain is present you've probably incorrectly generated the sha256 of the certificate.

If the certificate chain is not present, you've incorrectly setup the certificate on your Elasticsearch node. If it's a plaintext ssl crt file, you need to go modify the public certificate on your Elasticsearch nodes and paste in the ca cert so that the public certificate file contains both the public cert and the ca cert.

If you're using a non plaintext certificate format you'll need to use openssl to embed the ca cert into the certificate.

You'll need to do this on each elasticsearch node

Regarding the fingerprint errors being printed, I've gone ahead and made an issue and drafted a PR to improve the warnings printed when using the ca trusted fingerprint setting: Improve trusted_ca_fingerprint warnings and fix tests by strawgate · Pull Request #285 · elastic/elastic-agent-libs · GitHub