Fleet server - dial tcp connection refused

Elastic Stack Elastic Agent
1

Hi,
I have docker compose file for elasticsearch, kibana, elastic-agent and fleet server (version 8.17.0). Elasticsearch works well with Kibana. The problem is with starting fleet server where the starting procedure fails with

{"log.level":"error","@timestamp":"2025-01-08T00:24:29.659Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed fleet-server-default-fleet-server (STARTING->FAILED): Error - failed version compatibility check with elasticsearch: dial tcp [::1]:9200: connect: connection refused","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"FAILED","old_state":"STARTING"},"ecs.version":"1.6.0"}

The interresting thing is, that even I changed the
ELASTICSEARCH_HOST to port 9300, the log still contains tcp [::1]:9200. So I don't know where comes the url to elasticsearch from.

Here is my docker compose definition part:

elastic-agent:
  depends_on:
    kibana:
      condition: service_healthy
  image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
  restart: always
  volumes:
    - certs:/certs
    - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
    - "/var/run/docker.sock:/var/run/docker.sock:ro"
  user: root
  environment:
    - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
    - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
    - FLEET_CA=/certs/ca/ca.crt
    - FLEET_ENROLL=1
    - FLEET_URL=https://fleet-server:${FLEET_PORT}
    - KIBANA_FLEET_CA=/certs/ca/ca.crt
    - KIBANA_FLEET_USERNAME=elastic
    - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
    - KIBANA_FLEET_HOST=http://kibana:${KIBANA_PORT}
    - ELASTICSEARCH_HOST=https://es01:9300
  healthcheck:
    test:
      [ 
        "CMD-SHELL", 
        "curl -s --cacert /certs/ca/ca.crt https://fleet-server:${FLEET_PORT}/api/status | grep HEALTHY 2>&1 >/dev/null",
      ]
    interval: 10s
    timeout: 10s
    retries: 120

fleet-server:
  extends:
    service: elastic-agent
  depends_on:
    kibana:
      condition: service_healthy
    es01:
      condition: service_healthy
  volumes:
    - fleetserverdata:/usr/share/elastic-agent
    - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
    - "/proc:/hostfs/proc:ro"
    - "/:/hostfs:ro"
  ports:
    - ${FLEET_PORT}:8220
    - ${APMSERVER_PORT}:8200
  environment:
    - FLEET_INSECURE=true
    - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
    - ELASTICSEARCH_HOST=https://es01:9300
    #- FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
    - FLEET_SERVER_ELASTICSEARCH_USERNAME=elastic
    - FLEET_SERVER_ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}
    - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
    - FLEET_SERVER_ENABLE=1
    - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
    - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
    - FLEET_SERVER_INSECURE_HTTP=true
    - FLEET_SERVER_POLICY_ID=fleet-server-policy
    - KIBANA_FLEET_SETUP=1
  healthcheck:
    test:
      [ 
        "CMD-SHELL", 
        "curl -s --cacert /certs/ca/ca.crt https://localhost:${FLEET_PORT}/api/status | grep HEALTHY 2>&1 >/dev/null",
      ]
    interval: 10s
    timeout: 10s
    retries: 120
2

Hello and welcome,

You need to share your full compose with your Elasticsearch container configuration as well.

Also, Elasticsearch uses port 9200 for http and 9300 for transport, which is the communication between nodes.

It is not clear why you are using por 9300 in configuration in both your Kibana and Elastic Agent containers.

3

OK, the 9300 port was used just for the test if the variable is propagated into the fleet server log. But it didn't.

So here is the original complete docker-compose.yml file:

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
          cat config/certs/ca/ca.crt config/certs/ca/ca.key > config/certs/ca/ca.pem
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: kibana\n"\
          "    dns:\n"\
          "      - kibana\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: fleet-server\n"\
          "    dns:\n"\
          "      - fleet-server\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    hostname: elasticsearch
    ports:
      - ${ES_PORT}:9200
    volumes:
      - certs:/usr/share/elasticsearch/config/certs
      - esdata01:/usr/share/elasticsearch/data
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - xpack.ml.use_auto_machine_memory_percent=true
      - ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
      - ELASTIC_APM_ENVIRONMENT=${STACK_APM_ENVIRONMENT} 
      - APMSERVER_PORT=${APMSERVER_PORT} 
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:${ES_PORT} | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120
 
  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibanadata:/usr/share/kibana/data
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:${ES_PORT}
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
      - FLEET_PORT=${FLEET_PORT}
      - APMSERVER_PORT=${APMSERVER_PORT}
      - XPACK_SECURITY_ENCRYPTIONKEY=${XPACK_ENCRYPTION_KEY}
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=${XPACK_ENCRYPTION_KEY}
      - XPACK_REPORTING_ENCRYPTIONKEY=${XPACK_ENCRYPTION_KEY}
      - XPACK_REPORTING_KIBANASERVER_HOSTNAME=localhost
      - ELASTIC_APM_SECRET_TOKEN=${ELASTIC_APM_SECRET_TOKEN}
      - ELASTIC_APM_ENVIRONMENT=${STACK_APM_ENVIRONMENT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:${KIBANA_PORT} | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  elastic-agent:
    depends_on:
      kibana:
        condition: service_healthy
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    restart: always
    volumes:
      - certs:/certs
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    user: root
    environment:
      - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - FLEET_CA=/certs/ca/ca.crt
      - FLEET_ENROLL=1
      - FLEET_URL=https://fleet-server:${FLEET_PORT}
      - KIBANA_FLEET_CA=/certs/ca/ca.crt
      - KIBANA_FLEET_USERNAME=elastic
      - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_FLEET_HOST=http://kibana:${KIBANA_PORT}
      - ELASTICSEARCH_HOST=https://es01:${ES_PORT}
    healthcheck:
      test:
        [ 
          "CMD-SHELL", 
          "curl -s --cacert /certs/ca/ca.crt https://fleet-server:${FLEET_PORT}/api/status | grep HEALTHY 2>&1 >/dev/null",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  fleet-server:
    extends:
      service: elastic-agent
    depends_on:
      kibana:
        condition: service_healthy
      es01:
        condition: service_healthy
    volumes:
      - fleetserverdata:/usr/share/elastic-agent
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    ports:
      - ${FLEET_PORT}:8220
      - ${APMSERVER_PORT}:8200
    environment:
      - FLEET_INSECURE=true
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:${ES_PORT}
      - FLEET_SERVER_ELASTICSEARCH_USERNAME=elastic
      - FLEET_SERVER_ELASTICSEARCH_PASSWORD=${ELASTIC_PASSWORD}
      - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
      - FLEET_SERVER_ENABLE=1
      - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
      - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
      - FLEET_SERVER_INSECURE_HTTP=true
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - KIBANA_FLEET_SETUP=1
    healthcheck:
      test:
        [ 
          "CMD-SHELL", 
          "curl -s --cacert /certs/ca/ca.crt https://localhost:${FLEET_PORT}/api/status | grep HEALTHY 2>&1 >/dev/null",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local 
  fleetserverdata:
    driver: local

and the .env file:

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=ElasticXYZ

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=KibanaXYZ

# Version of Elastic products
STACK_VERSION=8.17.0

# Set the cluster name
CLUSTER_NAME=docker-cluster

# License
LICENSE=basic

# Port to expose Elasticsearch
ES_PORT=9200

# Port to expose Kibana
KIBANA_PORT=5601

# Port to expose Fleet server
FLEET_PORT=8220

# Port to expose APM
APMSERVER_PORT=8200

# APM Secret Token for POC environments only
ELASTIC_APM_SECRET_TOKEN=supersecrettoken

# AMP Environment for Elastic Stack monitoring
STACK_APM_ENVIRONMENT=Elastic_Stack

# SAMPLE Predefined Key
XPACK_ENCRYPTION_KEY=a34b38c3d14956121ff2170e5030f471551370178f43e5626eec58b04a30fae2

It is ready for running, can you pls check it?

4

Could not replicate your same issue.

I needed to make some changes in your compose, like bind mounting the /certs paht into your fleet container, which you did not have in your docker compose.

The Fleet Server started, but it wouldn't work because you are missing extra configurations, you are telling the fleet server to use the fleet-server-policy, but this policy does not work, so it will not be able to fully start and will be in a loop trying to join this policy.

fleet-server-1 | {"log.level":"info","@timestamp":"2025-01-08T12:38:02.261Z","message":"Waiting on policy with Fleet Server integration: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","service.type":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"}

You need to create the policy first as mentioned in the documentation.

I suggest that you check this repository of a working single node cluster with fleet and adapt to your use case: GitHub - peasead/elastic-container: Stand up a simple Elastic container with Kibana, Fleet, and the Detection Engine

5

Thank you for your reply.
I don't know where the problem exactly was, but maybe it was some mismatch in old volumes data folders.

I made a lot of changes and then I restarted docker-compose but I didn't deleted the old volumes.

So for now, after deleting volumes and recreated containers it starts work.