Fleet Data streams is No data streams

Hello every one!

I'm having some issues, here are my installation steps

Install Elasticsearch with Docker on sever A.

.env

# Password for the 'elastic' user (at least 6 characters)
ELASTIC_PASSWORD=urpassword

# Password for the 'kibana_system' user (at least 6 characters)
KIBANA_PASSWORD=urpassword

# Version of Elastic products
STACK_VERSION=8.4.2

# Set the cluster name
CLUSTER_NAME=docker-cluster

# Set to 'basic' or 'trial' to automatically start the 30-day trial
LICENSE=basic
#LICENSE=trial

# Port to expose Elasticsearch HTTP API to the host
ES_PORT=9200

# Port to expose Kibana to the host
KIBANA_PORT=5601
#KIBANA_PORT=80

# Increase or decrease based on the available host memory (in bytes)
MEM_LIMIT=1073741824

# Project namespace (defaults to the current folder name if not set)
#COMPOSE_PROJECT_NAME=myproject

docke-compose.yml

version: "2.2"

services:
  setup:
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
    user: "0"
    command: >
      bash -c '
        if [ x${ELASTIC_PASSWORD} == x ]; then
          echo "Set the ELASTIC_PASSWORD environment variable in the .env file";
          exit 1;
        elif [ x${KIBANA_PASSWORD} == x ]; then
          echo "Set the KIBANA_PASSWORD environment variable in the .env file";
          exit 1;
        fi;
        if [ ! -f config/certs/ca.zip ]; then
          echo "Creating CA";
          bin/elasticsearch-certutil ca --silent --pem -out config/certs/ca.zip;
          unzip config/certs/ca.zip -d config/certs;
        fi;
        if [ ! -f config/certs/certs.zip ]; then
          echo "Creating certs";
          echo -ne \
          "instances:\n"\
          "  - name: es01\n"\
          "    dns:\n"\
          "      - es01\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es02\n"\
          "    dns:\n"\
          "      - es02\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          "  - name: es03\n"\
          "    dns:\n"\
          "      - es03\n"\
          "      - localhost\n"\
          "    ip:\n"\
          "      - 127.0.0.1\n"\
          > config/certs/instances.yml;
          bin/elasticsearch-certutil cert --silent --pem -out config/certs/certs.zip --in config/certs/instances.yml --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key;
          unzip config/certs/certs.zip -d config/certs;
        fi;
        echo "Setting file permissions"
        chown -R root:root config/certs;
        find . -type d -exec chmod 750 \{\} \;;
        find . -type f -exec chmod 640 \{\} \;;
        echo "Waiting for Elasticsearch availability";
        until curl -s --cacert config/certs/ca/ca.crt https://es01:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
        echo "Setting kibana_system password";
        until curl -s -X POST --cacert config/certs/ca/ca.crt -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" https://es01:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
        echo "All done!";
      '
    healthcheck:
      test: ["CMD-SHELL", "[ -f config/certs/es01/es01.crt ]"]
      interval: 1s
      timeout: 5s
      retries: 120

  es01:
    depends_on:
      setup:
        condition: service_healthy
    image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
    volumes:
      - ./certs:/usr/share/elasticsearch/config/certs
      - ./esdata01:/usr/share/elasticsearch/data
    ports:
      - ${ES_PORT}:9200
    environment:
      - node.name=es01
      - cluster.name=${CLUSTER_NAME}
      - cluster.initial_master_nodes=es01
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
      - bootstrap.memory_lock=true
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=certs/es01/es01.key
      - xpack.security.http.ssl.certificate=certs/es01/es01.crt
      - xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.http.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.key=certs/es01/es01.key
      - xpack.security.transport.ssl.certificate=certs/es01/es01.crt
      - xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.license.self_generated.type=${LICENSE}
      - ES_JAVA_OPTS:"-Xms2g -Xmx2g"
    mem_limit: ${MEM_LIMIT}
    ulimits:
      memlock:
        soft: -1
        hard: -1
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s --cacert config/certs/ca/ca.crt https://localhost:9200 | grep -q 'missing authentication credentials'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

  

  kibana:
    depends_on:
      es01:
        condition: service_healthy
    image: docker.elastic.co/kibana/kibana:${STACK_VERSION}
    volumes:
      - ./certs:/usr/share/kibana/config/certs
      # - ./kibanadata:/usr/share/kibana/data
      - ./kibanadata/kibana.yml:/usr/share/kibana/config/kibana.yml:ro,Z
    ports:
      - ${KIBANA_PORT}:5601
    environment:
      - SERVERNAME=kibana
      - ELASTICSEARCH_HOSTS=https://es01:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=${KIBANA_PASSWORD}
      - ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt
    mem_limit: ${MEM_LIMIT}
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -s -I http://localhost:5601 | grep -q 'HTTP/1.1 302 Found'",
        ]
      interval: 10s
      timeout: 10s
      retries: 120

volumes:
  certs:
    driver: local
  esdata01:
    driver: local
  kibanadata:
    driver: local

Then run

sudo docker compose up -d

Then copy the certs.zip and ca.zip from ./certs/

Unzip certs.zip and ca.zip to get ca.crt , es01.crt , es01.key.

Keep file to ready install Fleet Server.

Next step, Login Kibana by username "elastic" pwd "urpassword"

Into kibana, First step to Fleet page.

  1. Select setting tab.
  2. Fleet server hosts -> Edit hosts -> Enter Server B IP ex: https://10.220.0.20:8220
  3. Save and apply setting.
  4. Outputs -> Add output -> host -> Enter Server A IP ex: https://10.220.0.10:9200
  5. Save and apply setting.

Now! Is time to install Fleet Server.

Copy the "es01.key" and "es01.crt" and "ca.crt" to Server B. patch ex: /var/certs/ca.crt , es01.crt , es01.key.

At Fleet page, click Add a Fleet Server.

  1. select Advanced
  2. Select a policy for Fleet Server : use default "Fleet Server policy".
  3. Choose a deployment mode for security : choose Production.
  4. Add your Fleet Server host : select Server B IP , and add host.
  5. Generate a service token.
  6. Get code and run it on server B.
sudo ./elastic-agent install --url=https://10.220.0.20:8220 \
  --fleet-server-es=https://10.220.0.10:9200 \
  --fleet-server-service-token=<token> \
  --fleet-server-policy=fleet-server-policy \
  --certificate-authorities=/var/certs/ca.crt \
  --fleet-server-es-ca=/var/certs/ca.crt \
  --fleet-server-cert=/var/certs/es01.crt \
  --fleet-server-cert-key=/var/certs/es01.key

This code is install success and status is Healthy at Fleet page.

But in Data streams is no data streams.

How can I check any errors??

Check Fleet Server state

curl -f --insecure https://10.220.0.20:8220/api/status

return  {"name":"fleet-server","status":"HEALTHY"}

Check elasticsearch state

curl -f --insecure -u elastic:urpassword https://10.220.0.10:9200/_cat/health

return  1665654035 09:40:35 docker-cluster green 1 1 24 24 0 0 0 0 - 100.0%

On server B run

sudo elastic-agent inspect output --output default --program filebeat

The last line is return

output:
  elasticsearch:
    api_key: api_keyapi_keyapi_keyapi_key
    hosts:
    - https://10.220.0.10:9200

The actual datastream won't show up until there is data in it