Fleet Server Setup failing

Elastic Stack Elastic Agent
, ,
1

Hello,

Trying to enroll fleet in docker env and the enroll is not happening with the below continuous log messages.

[root@eicillp949 ~]# docker logs --since 10m -f elastic-agent
Policy selected for enrollment:  fleet-server-policy
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.675Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":597},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.951Z","log.origin":{"file.name":"cmd/run.go","file.line":236},"message":"APM instrumentation disabled","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.952Z","log.origin":{"file.name":"application/application.go","file.line":49},"message":"Gathered system information","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.973Z","log.origin":{"file.name":"application/application.go","file.line":55},"message":"Detected available inputs and outputs","log":{"source":"elastic-agent"},"inputs":["syncgateway/metrics","prometheus/metrics","docker","entity-analytics","azure/metrics","cloudfoundry/metrics","http/metrics","azure-blob-storage","container","docker/metrics","iis/metrics","apm","aws-s3","netflow","tcp","udp","beat/metrics","etcd/metrics","gcs","journald","audit/system","pf-elastic-symbolizer","enterprisesearch/metrics","postgresql/metrics","zookeeper/metrics","pf-host-agent","endpoint","cel","osquery","linux/metrics","jolokia/metrics","sql/metrics","fleet-server","mqtt","cloudfoundry","o365audit","unix","mongodb/metrics","nginx/metrics","rabbitmq/metrics","synthetics/tcp","cloud_defend/control","pf-elastic-collector","azure-eventhub","cometd","kibana/metrics","redis/metrics","haproxy/metrics","nats/metrics","filestream","cloudbeat/vuln_mgmt_aws","aws-cloudwatch","aws/metrics","synthetics/icmp","cloudbeat/cis_k8s","cloudbeat/cis_aws","activemq/metrics","vsphere/metrics","audit/auditd","audit/file_integrity","kubernetes/metrics","uwsgi/metrics","awsfargate/metrics","containerd/metrics","packet","http_endpoint","kafka","syslog","statsd/metrics","traefik/metrics","synthetics/http","cloudbeat","mysql/metrics","system/metrics","gcp/metrics","memcached/metrics","httpjson","log","windows/metrics","stan/metrics","redis","winlog","apache/metrics","kafka/metrics","synthetics/browser","gcp-pubsub","mssql/metrics","oracle/metrics","lumberjack","elasticsearch/metrics","logstash/metrics","cloudbeat/cis_eks"],"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.973Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":54},"message":"Capabilities file not found in /usr/share/elastic-agent/state/capabilities.yml","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:02.973Z","log.origin":{"file.name":"application/application.go","file.line":61},"message":"Determined allowed capabilities","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.118Z","log.origin":{"file.name":"application/application.go","file.line":138},"message":"Parsed configuration and determined agent is in Fleet Server bootstrap mode","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.123Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":74},"message":"Starting stats endpoint","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.123Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":76},"message":"Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.125Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.125Z","log.origin":{"file.name":"upgrade/upgrade.go","file.line":99},"message":"Source URI reset from \"https://artifacts.elastic.co/downloads/\" to \"https://artifacts.elastic.co/downloads/\"","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.226Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":821},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.571Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":821},"message":"Updating running component model","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.571Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":970},"message":"Spawned new component fleet-server-default: Starting: spawned pid '41'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.571Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":970},"message":"Spawned new unit fleet-server-default: Starting: spawned pid '41'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default","type":"output","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.571Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":970},"message":"Spawned new unit fleet-server-default-fleet-server: Starting: spawned pid '41'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"STARTING"},"unit":{"id":"fleet-server-default-fleet-server","type":"input","state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.598Z","message":"Boot fleet-server","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"exe":"/usr/share/elastic-agent/data/elastic-agent-4ac18b/components/fleet-server","args":["--agent-mode","-E","logging.level=debug","-E","logging.to_stderr=true","-E","path.data=/usr/share/elastic-agent/state/data/run/fleet-server-default"],"ppid":29,"ecs.version":"1.6.0","service.name":"fleet-server","version":"8.8.1","commit":"672f072a","pid":41,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.600Z","message":"starting communication connection back to Elastic Agent","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","@timestamp":"2023-08-22T13:25:03.6Z","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.609Z","log.origin":{"file.name":"coordinator/coordinator.go","file.line":976},"message":"Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '41'","log":{"source":"elastic-agent"},"component":{"id":"fleet-server-default","state":"HEALTHY","old_state":"STARTING"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.612Z","message":"Setting cache config options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"cfg":{"APIKeyJitter":300000000000,"APIKeyTTL":900000000000,"ActionTTL":300000000000,"ArtifactTTL":86400000000000,"EnrollKeyTTL":60000000000,"MaxCost":52428800,"NumCounters":500000},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.614Z","message":"Starting","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.614Z","message":"initial server configuration","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","new":{"Fleet":{"Agent":{"ID":"","Logging":{"Level":"info"},"Version":""},"Host":{"ID":"","Name":""}},"HTTP":{"Enabled":false,"Host":"localhost","Port":5066,"SecurityDescriptor":"","User":""},"Inputs":[{"Cache":{"APIKeyJitter":0,"APIKeyTTL":0,"ActionTTL":0,"ArtifactTTL":0,"EnrollKeyTTL":0,"MaxCost":0,"NumCounters":0},"Monitor":{"FetchSize":0,"PollTimeout":0},"Policy":{"ID":""},"Server":{"Bulk":{"FlushInterval":250000000,"FlushMaxPending":8,"FlushThresholdCount":2048,"FlushThresholdSize":1048576},"CompressionLevel":1,"CompressionThresh":1024,"GC":{"CleanupAfterExpiredInterval":"30d","ScheduleInterval":3600000000000},"Host":"0.0.0.0","Instrumentation":{"APIKey":"","APIKeyPath":"","Enabled":false,"Environment":"","GlobalLabels":"","Hosts":null,"SecretToken":"","SecretTokenPath":"","TLS":{"ServerCA":"","ServerCertificate":"","SkipVerify":false},"TransactionSampleRate":""},"InternalPort":8221,"Limits":{"AckLimit":{"Burst":100,"Interval":10000000,"Max":50,"MaxBody":2097152},"ArtifactLimit":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"CheckinLimit":{"Burst":1000,"Interval":1000000,"Max":0,"MaxBody":1048576},"EnrollLimit":{"Burst":100,"Interval":10000000,"Max":50,"MaxBody":524288},"MaxAgents":0,"MaxConnections":0,"MaxHeaderByteSize":8192,"PolicyThrottle":5000000,"StatusLimit":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"UploadChunkLimit":{"Burst":10,"Interval":3000000,"Max":5,"MaxBody":4194304},"UploadEndLimit":{"Burst":5,"Interval":2000000000,"Max":2,"MaxBody":1024},"UploadStartLimit":{"Burst":8,"Interval":3000000000,"Max":3,"MaxBody":5242880}},"Port":8220,"Profiler":{"Bind":"localhost:6060","Enabled":false},"Runtime":{"GCPercent":0,"MemoryLimit":0},"TLS":{"CASha256":null,"CAs":null,"Certificate":{"Certificate":"/usr/tmp/elastic-agent/config/fleet-cert.crt","Key":"[redacted]","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"ClientAuth":0,"CurveTypes":null,"Enabled":null,"VerificationMode":"full","Versions":null},"Timeouts":{"CheckinJitter":30000000000,"CheckinLongPoll":300000000000,"CheckinMaxPoll":3600000000000,"CheckinTimestamp":30000000000,"Idle":30000000000,"Read":60000000000,"ReadHeader":5000000000,"Write":600000000000}},"Type":""}],"Logging":{"Files":null,"Level":"info","Pretty":false,"ToFiles":true,"ToStderr":true},"Output":{"Elasticsearch":{"Headers":null,"Hosts":["monitor.pulse.com:9200"],"MaxConnPerHost":128,"MaxContentLength":104857600,"MaxRetries":3,"Path":"","Protocol":"https","ProxyDisable":false,"ProxyHeaders":{},"ProxyURL":"","ServiceToken":"[redacted]","ServiceTokenPath":"","TLS":{"CASha256":null,"CATrustedFingerprint":"","CAs":["/usr/tmp/elastic-agent/config/elasticsearch_ca.crt"],"Certificate":{"Certificate":"","Key":"","Passphrase":"","PassphrasePath":""},"CipherSuites":null,"CurveTypes":null,"Enabled":null,"Renegotiation":"never","VerificationMode":"full","Versions":null},"Timeout":90000000000},"Extra":null}},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.614Z","message":"starting server on configuration change","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.654Z","message":"elasticsearch cluster info","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"cluster.uuid":"LeGLEfC0SeirLmx3zqTl7w","cluster.version":"8.8.1","service.name":"fleet-server","cluster.maxConnsPersHost":128,"cluster.addr":["monitor.pulse.com:9200"],"cluster.name":"es-monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.655Z","message":"Run bulker with options","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"opts":{"apikeyMaxParallel":120,"apikeyMaxReqSize":104857600,"blockQueueSz":32,"flushInterval":250,"flushThresholdCnt":2048,"flushThresholdSz":1048576,"maxPending":8},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.655Z","message":"Elasticsearch compatibility check successful","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","fleet_version":"8.8.1","elasticsearch_version":"8.8.1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.658Z","message":"migration AgentMetadata done","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"fleet.migration.batches":0,"fleet.migration.retries.search":0,"fleet.migration.total.duration":2.092724,"fleet.migration.es.took":0,"fleet.migration.updated":0,"fleet.migration.deleted":0,"fleet.migration.version_conflicts":0,"service.name":"fleet-server","fleet.migration.name":"AgentMetadata","fleet.migration.retries.bulk":0,"ecs.version":"1.6.0","fleet.migration.noops":0,"fleet.migration.total.count":0,"fleet.migration.es.timed_out":false,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.659Z","message":"migration AgentOutputs done","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","fleet.migration.batches":0,"fleet.migration.noops":0,"fleet.migration.retries.search":0,"fleet.migration.total.count":0,"service.name":"fleet-server","fleet.migration.es.took":0,"fleet.migration.updated":0,"fleet.migration.deleted":0,"fleet.migration.version_conflicts":0,"fleet.migration.retries.bulk":0,"fleet.migration.total.duration":1.616916,"fleet.migration.name":"AgentOutputs","fleet.migration.es.timed_out":false,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"elasticsearch cluster info","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","cluster.maxConnsPersHost":128,"cluster.name":"es-monitor","cluster.uuid":"LeGLEfC0SeirLmx3zqTl7w","cluster.addr":["monitor.pulse.com:9200"],"cluster.version":"8.8.1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"run policy monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ctx":"policy agent monitor","throttle":5,"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"upload limits","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","limits":{"Burst":25,"Interval":5000000,"Max":50,"MaxBody":0},"maxFileSize":104857600,"ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","index":".fleet-policies","ctx":"index monitor","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-08-22T13:25:03.671Z","message":"missing config fleet.agent.id; acceptable until Elastic Agent has enrolled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ctx":"policy leader manager","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"starting index monitor","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","index":".fleet-actions","ctx":"index monitor","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.671Z","message":"server hard connection limiter disabled","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.672Z","message":"Listening on 0.0.0.0:8220","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.673Z","message":"Listening on localhost:8221","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:03.928Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:04.681Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":806},"message":"Fleet Server - Waiting on fleet-server input to be added to policy: fleet-server-policy","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:09.186Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:14.444Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:19.702Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:24.959Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:30.217Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:35.475Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:40.733Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:45.993Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"STARTING","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:51.251Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:25:56.508Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:01.765Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"STARTING","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:07.022Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:12.279Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"STARTING","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:17.538Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:22.795Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:28.052Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:33.311Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:38.569Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:43.827Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:49.084Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:54.343Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"state":"STARTING","ecs.version":"1.6.0","service.name":"fleet-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-22T13:26:59.600Z","message":"Waiting on fleet-server input to be added to policy: fleet-server-policy","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","state":"STARTING","@timestamp":"2023-08-22T13:26:59.6Z","ecs.version":"1.6.0"}

Anyone encountered/ aware of the situation? Any help is appreciated :slight_smile:

Thanks in advance.

2

Hi Sai,

Based on the logs, there seems to be fleet-server input missing from the fleet-server-policy. Can you check in agent policies that your Fleet Server policy has a Fleet Server integration?

3

@Julia_Bardi Fleet server integration is existing in the policy.

image
image1220×298 24.8 KB

Indeed I see the fleet host input is missing in the Policy but all the configuration is in place and I see possible reason why it's missing.

image

 "inputs": [
              {
                "id": "fleet-server-fleet_server-55eb72d0-40eb-11ee-a122-738377946509",
                "revision": 1,
                "name": "fleet_server-1",
                "type": "fleet-server",
                "data_stream": {
                  "namespace": "default"
                },
                "use_output": "default",
                "package_policy_id": "55eb72d0-40eb-11ee-a122-738377946509",
                "unused_key": "not_used",
                "meta": {
                  "package": {
                    "name": "fleet_server",
                    "version": "1.3.1"
                  }
                }
              }
            ]

Thanks for the response :slight_smile:

4

Correction: I don't see any reason why fleet server input is missing.

5

What you could check is the result of this ES query to see if there are multiple documents with the latest revision_idx (it is 7 on the screenshot):

GET .fleet-policies/_search?q=policy_id:<policy_id>
{
  "sort": [
    {
      "revision_idx": {
        "order": "desc"
      }
    }
  ]
}

If there are multiple documents, and one of them is missing inputs, it could be the reason of the issue. In that case you can update the policy (e.g. change description) to trigger a new revision, and try enrolling again.

6

Regards,
Sai

7

That info log about agent.id can be ignored. Is there anything else in the logs after this?

8

@Julia_Bardi

{"log.level":"info","@timestamp":"2023-08-25T14:24:10.862Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":787},"message":"Fleet Server - Running on policy with Fleet Server integration: 36325990-428f-11ee-a122-738377946509; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:11.645Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: https://<fleet_server>:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:11.868Z","log.origin":{"file.name":"cmd/run.go","file.line":300},"message":"signal \"terminated\" received","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:11.868Z","log.origin":{"file.name":"cmd/run.go","file.line":312},"message":"Shutting down Elastic Agent and sending last events...","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:11.868Z","message":"On signal","component":{"binary":"fleet-server","dataset":"elastic_agent.fleet_server","id":"fleet-server-default","type":"fleet-server"},"log":{"source":"fleet-server-default"},"ecs.version":"1.6.0","service.name":"fleet-server","sig":"terminated","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:12.069Z","log.origin":{"file.name":"cmd/run.go","file.line":318},"message":"Shutting down completed.","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-08-25T14:24:12.069Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":79},"message":"Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp <fleet_server>:8220 connect: connection refused
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.8/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.8/fleet-troubleshooting.html

I could see these but after these logs I see continuous logging saying agent.id is missing.

9

Which command are you using to enroll the fleet-server?
This message is strange, it sounds like trying to enroll a regular agent, not fleet-server: Starting enrollment to URL: https://<fleet_server>:8220/"

10

@Julia_Bardi

Using Docker Compose and deploying via Ansible

container:
  image: docker.elastic.co/beats/elastic-agent
  tag: 8.8.1
  name: elastic-agent
  ports:
    - '6791:6791'
    - '8220:8220'
  volumes:
    - '/docker/elastic-agent/data:/usr/tmp/elastic-agent/data'
    - '/etc/elastic-agent:/usr/tmp/elastic-agent/config'
    - '/var/log/docker/elastic-agent:/usr/tmp/elastic-agent/logs'
  env:
    FLEET_SERVER_ENABLE: "1"
    FLEET_URL: 'https://<fleet_server_host_machine_IP>:8220'
    FLEET_SERVER_PORT: '8220'
    FLEET_SERVER_ELASTICSEARCH_HOST: 'https://monitor.eu:9200'
    FLEET_SERVER_SERVICE_TOKEN: '****'
    FLEET_SERVER_POLICY_ID: '36325990-428f-11ee-a122-738377946509'
    FLEET_SERVER_ELASTICSEARCH_CA: '/usr/tmp/elastic-agent/config/elasticsearch_ca.crt' 
    FLEET_SERVER_CERT: '/usr/tmp/elastic-agent/config/fleet-cert.crt'
    FLEET_SERVER_CERT_KEY: '/usr/tmp/elastic-agent/config/fleet-server.key'
11

Do you want to start a Fleet Server like this? Then you need additional envvars:

   - FLEET_SERVER_ENABLE=true
   - FLEET_SERVER_ELASTICSEARCH_HOST={elasticsearch-host}
   - FLEET_SERVER_SERVICE_TOKEN={service-token}

See the docs: Run Elastic Agent in a container | Fleet and Elastic Agent Guide [8.2] | Elastic

12

@Julia_Bardi I was refering Run Elastic Agent in a container | Fleet and Elastic Agent Guide [8.9] | Elastic as we are in latest version . In any case connection refused issue is now fixed. There was a problem in port binding and now the destination is reachable.

I have below error

Error: fail to enroll: fail to execute request to fleet-server: x509: certificate signed by unknown authority
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.8/fleet-troubleshooting.html
Error: enrollment failed: exit status 1

I thinks its more of how the certificates are generated as we have internal CA, Should get that checked.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.