am running a fresh docker based instance of elasticsearch, kibana and fleet-server (all in separate containers).
Now I am starting a fresh elastic-agent container which enrolls in fleet just fine but don't receive his configuration from fleet.
This is the full log of the agent:
{"log.level":"info","@timestamp":"2023-02-15T15:07:30.297Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Starting enrollment to URL: https://fleet-server:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:31.826Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":271},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.053Z","log.origin":{"file.name":"cmd/run.go","file.line":192},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.053Z","log.origin":{"file.name":"application/application.go","file.line":45},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.063Z","log.origin":{"file.name":"application/application.go","file.line":51},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["docker/metrics","vsphere/metrics","azure-blob-storage","syslog","udp","synthetics/tcp","cloudbeat","gcp-pubsub","redis","mssql/metrics","netflow","synthetics/browser","synthetics/http","synthetics/icmp","http/metrics","kafka/metrics","osquery","docker","journald","log","kubernetes/metrics","logstash/metrics","etcd/metrics","apm","fleet-server","system/metrics","stan/metrics","audit/file_integrity","aws-cloudwatch","aws-s3","httpjson","traefik/metrics","audit/system","endpoint","uwsgi/metrics","oracle/metrics","syncgateway/metrics","jolokia/metrics","elasticsearch/metrics","azure/metrics","activemq/metrics","apache/metrics","gcp/metrics","cel","http_endpoint","unix","winlog","aws/metrics","sql/metrics","nats/metrics","container","mqtt","beat/metrics","kibana/metrics","mysql/metrics","cloudfoundry/metrics","containerd/metrics","packet","kafka","tcp","mongodb/metrics","iis/metrics","nginx/metrics","zookeeper/metrics","redis/metrics","audit/auditd","azure-eventhub","cloudfoundry","gcs","o365audit","filestream","linux/metrics","awsfargate/metrics","prometheus/metrics","haproxy/metrics","rabbitmq/metrics","cloudbeat/cis_k8s","cloudbeat/cis_eks","postgresql/metrics","windows/metrics"],"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.063Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":54},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.063Z","log.origin":{"file.name":"application/application.go","file.line":57},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.328Z","log.origin":{"file.name":"application/application.go","file.line":111},"message":"Parsed configuration and determined agent is managed by Fleet","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.329Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.329Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-02-15T15:07:32.943Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":145},"message":"Fleet gateway started","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
The configuration of the container is as follows:
elastic-agent:
container_name: elk_elastic-agent
hostname: elastic-agent
build:
context: elastic-agent/
args:
ELK_VERSION: $ELK_VERSION
restart: unless-stopped
user: root
environment:
FLEET_CA: '/ca.crt'
ELASTICSEARCH_CA: '/ca.crt'
KIBANA_CA: '/ca.crt'
ELK_VERSION: ${ELK_VERSION}
KIBANA_HOST: "https://kibana:5601"
ELASTICSEARCH_USERNAME: ${ELASTIC_USERNAME}
ELASTICSEARCH_PASSWORD: ${ELASTIC_PASSWORD}
ELASTICSEARCH_HOSTS: "https://elasticsearch:9200"
FLEET_URL: "https://fleet-server:8220"
FLEET_ENROLLMENT_TOKEN: '<my-token>'
FLEET_ENROLL: 1
SSL.CERTIFICATE_AUTHORITIES: '/ca.crt'
ELASTIC_AGENT_TAGS: 'ELK, ELK-Monitoring'
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
- /proc:/hostfs/proc:ro
- /:/hostfs:ro
secrets:
- source: ca.crt
target: /ca.crt
networks:
- elk
depends_on:
- fleet-server
running elastic-agent status within the container delivers this:
root@elastic-agent:/usr/share/elastic-agent# elastic-agent status
State: STARTING
Message: Waiting for initial configuration and composable variables
Components: (none)
The standard image is amended to include the root certificate at the OS level using this Dockerfile:
ARG ELK_VERSION
FROM docker.elastic.co/beats/elastic-agent:${ELK_VERSION}
USER root
ADD cert/ca.crt /usr/local/share/ca-certificates/ca.crt
RUN chmod 644 /usr/local/share/ca-certificates/ca.crt && update-ca-certificates
RUN chown -R 0:0 /usr/share/elastic-agent/data
This configuration was working in my first implementation (on 8.6.0) which was my first experience with ES. Now I wanted to recreate the setup in a more controlled and cleaner way. But reusing the previously working asses surfaces new issues which havent been there previously
Any ideas where I can look to identify the missing bit for the agent to receive is configuration?