Elastic Agent show `Error dialing x509: certificate signed by unknown authority` but it is healthy in fleet

I am building elastic stack for testing, which uses self-signed certificate. And i use docker compose to deploy them.

Here is part of compose

  efleet:
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elastic-agent/certs
    environment:
      - FLEET_ENROLLMENT_TOKEN=<TOEKN>
      - FLEET_ENROLL=1
      - FLEET_URL=https://efleet:8220
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_HOST=0.0.0.0
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_SERVICE_TOKEN=<TOEKN>
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - FLEET_CA=/usr/share/elastic-agent/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_CA=/usr/share/elastic-agent/certs/ca/ca.crt
      - FLEET_SERVER_CERT=/usr/share/elastic-agent/certs/efleet/efleet.crt
      - FLEET_SERVER_CERT_KEY=/usr/share/elastic-agent/certs/efleet/efleet.key
    deploy:
      resources:
        limits:
          memory: ${MEM_LIMIT}
    networks:
      - fleet

  agent01:
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elastic-agent/certs
    environment:
      - FLEET_ENROLLMENT_TOKEN=<TOKEN>
      - FLEET_ENROLL=1
      - FLEET_URL=https://efleet:8220
      - FLEET_SERVER_HOST=0.0.0.0
      - FLEET_CA=/usr/share/elastic-agent/certs/ca/ca.crt
      - FLEET_INSECURE=true
    deploy:
      resources:
        limits:
          memory: ${MEM_LIMIT}
    networks:
      - fleet

I built up fleet and agent server, and kibana fleet shows them are healthy, but no data coming from agent. Then i watch docker log

{
    "log.level": "error",
    "@timestamp": "2023-04-06T12:36:39.768Z",
    "message": "Error dialing x509: certificate signed by unknown authority",
    "component": {
        "binary": "metricbeat",
        "dataset": "elastic_agent.metricbeat",
        "id": "http/metrics-monitoring",
        "type": "http/metrics"
    },
    "log": {
        "source": "http/metrics-monitoring"
    },
    "log.origin": {
        "file.line": 38,
        "file.name": "transport/logging.go"
    },
    "address": "es04:9200",
    "ecs.version": "1.6.0",
    "log.logger": "esclientleg",
    "service.name": "metricbeat",
    "network": "tcp",
    "ecs.version": "1.6.0"
}

Note: i can successfully access es01-es04

Update: My fleet got certificate error, which is same as agent. I forget to check fleet log while im writing question

I use this script to create certificates

  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es04\n"\
          "    dns:\n"\
          "      - es04\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: efleet\n"\
          "    dns:\n"\
          "      - efleet\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: [ "CMD-SHELL", "[ -f config/certs/es01/es01.crt ]" ]
      interval: 1s
      timeout: 5s
      retries: 120
    networks:
      - fleet

Fleet error

{
    "log.level": "error",
    "@timestamp": "2023-04-06T12:59:44.123Z",
    "message": "Failed to connect to backoff(elasticsearch(https://es01:9200)): Get \"https://es01:9200\": x509: certificate signed by unknown authority",
    "component": {
        "binary": "metricbeat",
        "dataset": "elastic_agent.metricbeat",
        "id": "beat/metrics-monitoring",
        "type": "beat/metrics"
    },
    "log": {
        "source": "beat/metrics-monitoring"
    },
    "log.logger": "publisher_pipeline_output",
    "log.origin": {
        "file.line": 150,
        "file.name": "pipeline/client_worker.go"
    },
    "service.name": "metricbeat",
    "ecs.version": "1.6.0",
    "ecs.version": "1.6.0"
}

After i read relative answer then add fingerprint to Elasticsearch CA trusted fingerprint (optional) and ssl.certificate_authorities: ["/usr/share/elastic-agent/certs/ca/ca.crt"] to Advanced YAML configuration. Agent and Fleet run successfully, but i still want to ask if i can set those arguments before docker compose build up?

There are two way to specify CA path

The first one is to follow this instruction: Configure SSL/TLS for self-managed Fleet Servers | Fleet and Elastic Agent Guide [8.7] | Elastic, and add ssl.certificate_authorities: ["/usr/share/elastic-agent/certs/ca/ca.crt"] to Advanced YAML configuration

The second one is to configure elastic-agent.yml directly, you can find ssl.certificate_authorities
in this * SSL/TLS.

Note: this settings seems to be dynamic, Configuring Elasticsearch | Elasticsearch Guide [8.7] | Elastic

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.