Fleet Server - Error - x509: certificate signed by unknown authority

Hi everyone :slight_smile:

Im trying to make a test setup with a Ubuntu 20.04 , running with docker.

I use docker-compose to make elastic agent/fleet server, but i cant make it work with a self signed certificate for fleet, elasticsearch and kibana and by adding the insecure to my docker-compose file.

I use this guide for setting up Elasticsearch & kibana, and confirmed that it did work.

Running the Elastic Stack on Docker | Getting Started [7.14] | Elastic

For Fleet / Elastic agent i have used following for inspiration:

Fleet Server installation options - Elastic Stack / Kibana - Discuss the Elastic Stack

Quick start: Get logs, metrics, and uptime data into the Elastic Stack | Fleet User Guide [7.14] | Elastic

Run Elastic Agent in a container | Fleet User Guide [7.14] | Elastic

My docker-compose looks like this:

version: "3.7"
services:
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:7.14.0
    container_name: fleet01
    restart: always
    user: root
    environment:
      - FLEET_ENROLL=1
      - FLEET_URL="https://ip:port"
      - FLEET_SERVER_ENABLE=true
      - ELASTICSEARCH_HOST=https://ip:port
      - FLEET_SERVER_SERVICE_TOKEN=SECRET
      - FLEET_CA=$CERTS_DIR/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=$CERTS_DIR/ca/ca.crt
      - FLEET_SERVER_ES_CA=$CERTS_DIR/ca/ca.crt
      - FLEET_SERVER_CERT=$CERTS_DIR/fleet01/fleet01.crt
      - FLEET_SERVER_CERT_KEY=$CERTS_DIR/fleet01/fleet01.key
      - FLEET_SERVER_POLICY=72495520-03e7-11ec-8ffc-b16ddd514078
      - FLEET_INSECURE=1
      - FLEET_SERVER_INSECURE_HTTP=0
    volumes:
      - certs:$CERTS_DIR
    networks:
      - es_elastic
volumes:
  certs:
    external: true
    name: es_certs

networks:
  es_elastic:
    external: true
    name: es_elastic

Logs i get with command docker logs fleet01


2021-09-02T12:22:24.713Z        INFO    cmd/enroll_cmd.go:526   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-02T12:22:24.879Z        INFO    application/application.go:66   Detecting execution mode
2021-09-02T12:22:24.881Z        INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-02T12:22:25.000Z        INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-02T12:22:25.000Z        INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-09-02T12:22:25.000Z        INFO    [api]   api/server.go:64        Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)
2021-09-02T12:22:25.001Z        INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-09-02T12:22:25.002Z        INFO    stateresolver/stateresolver.go:48       New State ID is btYyflaV
2021-09-02T12:22:25.002Z        INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-09-02T12:22:25.513Z        INFO    log/reporter.go:40      2021-09-02T12:22:25Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-02T12:22:25.514Z        INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-09-02T12:22:25.717Z        INFO    cmd/enroll_cmd.go:701   Fleet Server - Starting
2021-09-02T12:22:26.534Z        ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
***2021-09-02T12:22:26.535Z        ERROR   log/reporter.go:36      2021-09-02T12:22:26Z - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - x509: certificate signed by unknown authority - type: 'ERROR' - sub_type: 'FAILED'***
2021-09-02T12:22:26.719Z        INFO    cmd/enroll_cmd.go:701   Fleet Server - Error - x509: certificate signed by unknown authority
2021-09-02T12:22:32.729Z        INFO    cmd/enroll_cmd.go:706   Fleet Server - Error - x509: certificate signed by unknown authority
2021-09-02T12:22:36.536Z        INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-09-02T12:22:36.536Z        INFO    log/reporter.go:40      2021-09-02T12:22:36Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'
2021-09-02T12:22:36.536Z        INFO    log/reporter.go:40      2021-09-02T12:22:36Z - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-02T12:22:36.536Z        INFO    log/reporter.go:40      2021-09-02T12:22:36Z - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'

Solved.. but ran into another problem.

I was missing FLEET_SERVER_ELASTICSEARCH=$CERTS_DIR/ca/ca.crt

The error i got is below, seems like there is a problem with a config file which is missing. i'll look into this, if anyone got the same problem or have a solution please write a comment. :slight_smile:

note on error. I did change with ip address from the original log generated with docker logs

2021-09-06T13:15:25.445Z        INFO    cmd/enroll_cmd.go:508   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-06T13:15:25.610Z        INFO    application/application.go:66   Detecting execution mode
2021-09-06T13:15:25.612Z        INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-06T13:15:26.029Z        INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-06T13:15:26.029Z        INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-09-06T13:15:26.029Z        INFO    [api]   api/server.go:64        Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.s                                                                                ock)
2021-09-06T13:15:26.030Z        INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-09-06T13:15:26.031Z        INFO    stateresolver/stateresolver.go:48       New State ID is yut-NVDG
2021-09-06T13:15:26.031Z        INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-09-06T13:15:26.525Z        INFO    log/reporter.go:40      2021-09-06T13:15:26Z - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-06T13:15:26.526Z        INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-09-06T13:15:27.450Z        INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-06T13:15:28.048Z        WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-09-06T13:15:28.048Z        INFO    log/reporter.go:40      2021-09-06T13:15:28Z - message: Application: fleet-server--7.14.1[]: State changed to DEGRADED: Running on policy with Fleet Server integration: 64c04440-0f13-11ec-9032-e581                                                                                7e7d572b; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'
2021-09-06T13:15:28.453Z        INFO    cmd/enroll_cmd.go:664   Fleet Server - Running on policy with Fleet Server integration: 64c04440-0f13-11ec-9032-e5817e7d572b; **missing config fleet.agent.id** (expected during bootstrap process)
2021-09-06T13:15:28.454Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-06T13:15:29.376Z        INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: ip:8220
2021-09-06T13:15:29.480Z        WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-06T13:15:29.481Z        WARN    cmd/enroll_cmd.go:407   Remote server is not ready to accept connections, will retry in a moment.
2021-09-06T13:16:29.482Z        INFO    cmd/enroll_cmd.go:414   Retrying enrollment to URL: my ip:8220
2021-09-06T13:16:29.584Z        WARN    cmd/enroll_cmd.go:407   Remote server is not ready to accept connections, will retry in a moment.