Hey,
I'm trying to deploy Fleet Server and Elastic Agent on Kubernetes using the quickstart manual:
Since I've deployed my stack under a different namespace, all I did was changing the namespace on the provided manifest. Here is my end result:
agent.yaml:
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent-test
namespace: elastic
spec:
version: 8.4.1
kibanaRef:
name: test
namespace: elastic
fleetServerRef:
name: fleet-server-test
namespace: elastic
mode: fleet
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
fleet.yaml
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server-test
namespace: elastic
spec:
version: 8.4.1
kibanaRef:
name: test
elasticsearchRefs:
- name: test
mode: fleet
fleetServerEnabled: true
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: elastic
cluster-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- pods
- nodes
- namespaces
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
cluster-role-binding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: elastic
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
elasticsearch.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: test
namespace: elastic
spec:
version: 8.4.1
http:
tls:
certificate:
secretName: elk-certs
auth:
fileRealm:
- secretName: user-balink-admin
- secretName: user-logstash-internal
- secretName: user-logstash-beats
roles:
- secretName: role-logstash-writer
- secretName: role-logstash-beats
updateStrategy:
changeBudget:
maxSurge: 3
maxUnavailable: 1
nodeSets:
- name: masters
count: 1
config:
node.roles: ["master", "ingest", "remote_cluster_client"]
xpack.ml.enabled: true
node.store.allow_mmap: false # FIXME Good for local use only because of memory limitations
# azure.client.default.endpoint_suffix: core.chinacloudapi.cn # java.lang.IllegalArgumentException: missing required setting [azure.client.default.account] for setting [azure.client.default.endpoint_suffix]
# bootstrap.memory_lock: true # ???
podTemplate:
spec:
initContainers:
- name: install-plugins
command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch analysis-icu
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: standard
- name: data
count: 1
config:
node.roles:
["data", "ingest", "ml", "transform", "remote_cluster_client"]
node.store.allow_mmap: false # FIXME Good for local use only because of memory limitations
# azure.client.default.endpoint_suffix: core.chinacloudapi.cn # java.lang.IllegalArgumentException: missing required setting [azure.client.default.account] for setting [azure.client.default.endpoint_suffix]
podTemplate:
spec:
initContainers:
- name: install-plugins
command:
- sh
- -c
- |
bin/elasticsearch-plugin install --batch analysis-icu
volumeClaimTemplates:
- metadata:
name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
storageClassName: standard
kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: test
namespace: elastic
spec:
version: 8.4.1
count: 1
elasticsearchRef:
name: test
config:
xpack.fleet.agents.elasticsearch.hosts:
["https://test-es-http.elastic.svc:9200"]
xpack.fleet.agents.fleet_server.hosts:
["https://fleet-server-test-agent-http.elastic.svc:8220"]
xpack.fleet.packages:
- name: system
version: latest
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
xpack.fleet.agentPolicies:
- name: Fleet Server on ECK policy
id: eck-fleet-server
is_default_fleet_server: true
namespace: elastic
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: fleet_server-1
namespace: elastic
id: fleet_server-1
package:
name: fleet_server
- name: Elastic Agent on ECK policy
id: eck-agent
namespace: elastic
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
is_default: true
package_policies:
- name: system-1
namespace: elastic
id: system-1
package:
name: system
Kibana and elastic deploys well. I can also see the policies and enrollment tokens generated.
But, the fleet server is crashing.
kubectl get pods
NAME READY STATUS RESTARTS AGE
test-es-data-0 1/1 Running 0 140m
test-es-masters-0 1/1 Running 0 140m
test-kb-dbd564db8-gcfhf 1/1 Running 0 108m
elastic-agent-test-agent-hftkb 1/1 Running 1 (15m ago) 16m
fleet-server-test-agent-58b5fc8fb7-cxxl5 1/1 Error 5 (2m35s ago) 15m
Here are the logs from the elastic agent:
kubectl logs elastic-agent-test-agent-hftkb
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Requesting service_token from Kibana.
Created service_token named: token-1663077416076
Performing setup of Fleet in Kibana
Policy selected for enrollment: eck-agent
{"log.level":"info","@timestamp":"2022-09-13T13:56:57.429Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://fleet-server-test-agent-http.elastic.svc:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:58.252Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":271},"message":"Elastic Agent might not be running; unable to trigger restart","ecs.version":"1.6.0"}
Successfully enrolled the Elastic Agent.
{"log.level":"info","@timestamp":"2022-09-13T13:56:58.458Z","log.origin":{"file.name":"cmd/run.go","file.line":202},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:58.460Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:58.560Z","log.origin":{"file.name":"application/application.go","file.line":102},"message":"Agent is managed by Fleet","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:58.560Z","log.origin":{"file.name":"capabilities/capabilities.go","file.line":59},"message":"capabilities file not found in /usr/share/elastic-agent/capabilities.yml","ecs.version":"1.6.0"}
I0913 13:56:59.491120 926 leaderelection.go:248] attempting to acquire leader lease elastic/elastic-agent-cluster-leader...
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.491Z","log.logger":"composable.providers.docker","log.origin":{"file.name":"docker/docker.go","file.line":44},"message":"Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.491Z","log.logger":"composable.providers.kubernetes","log.origin":{"file.name":"kubernetes/kubernetes.go","file.line":92},"message":"Kubernetes provider started for resource pod with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.496Z","log.logger":"composable.providers.kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":129},"message":"kubernetes: Node minikube discovered by in cluster pod node query","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.799Z","log.logger":"composable.providers.kubernetes","log.origin":{"file.name":"kubernetes/kubernetes.go","file.line":92},"message":"Kubernetes provider started for resource node with node scope","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.799Z","log.logger":"composable.providers.kubernetes","log.origin":{"file.name":"kubernetes/util.go","file.line":122},"message":"kubernetes: Using node minikube provided in the config","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.903Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.903Z","log.origin":{"file.name":"application/managed_mode.go","file.line":323},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T13:56:59.903Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
I0913 13:57:15.964549 926 leaderelection.go:258] successfully acquired lease elastic/elastic-agent-cluster-leader
{"log.level":"error","@timestamp":"2022-09-13T13:59:34.275Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":293},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://fleet-server-test-agent-http.elastic.svc:8220/api/fleet/agents/afcdedc3-e064-4350-b5b2-77cd705b9281/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-13T14:01:59.686Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":293},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://fleet-server-test-agent-http.elastic.svc:8220/api/fleet/agents/afcdedc3-e064-4350-b5b2-77cd705b9281/checkin?\": EOF","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:05:39.091Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"component gateway-d037a83f: checkin failed: fail to checkin to fleet-server: Post \\\"https://fleet-server-test-agent-http.elastic.svc:8220/api/fleet/agents/afcdedc3-e064-4350-b5b2-77cd705b9281/checkin?\\\": EOF\"","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-13T14:07:15.078Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":293},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://fleet-server-test-agent-http.elastic.svc:8220/api/fleet/agents/afcdedc3-e064-4350-b5b2-77cd705b9281/checkin?\": EOF","ecs.version":"1.6.0"}
Logs from the crashing fleet server:
kubectl logs fleet-server-test-agent-58b5fc8fb7-cxxl5
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Performing setup of Fleet in Kibana
{"log.level":"info","@timestamp":"2022-09-13T14:11:44.882Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":587},"message":"Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:45.271Z","log.origin":{"file.name":"cmd/run.go","file.line":202},"message":"APM instrumentation disabled","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:45.272Z","log.origin":{"file.name":"application/application.go","file.line":78},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:45.358Z","log.origin":{"file.name":"application/application.go","file.line":98},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:45.956Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":759},"message":"Waiting for Elastic Agent to start Fleet Server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.156Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":68},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.156Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":134},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.156Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":70},"message":"Metrics endpoint listening on: /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock (configured: unix:///usr/share/elastic-agent/state/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.157Z","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":144},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.158Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is 8dhGi5qu","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.158Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:46.367Z","log.origin":{"file.name":"operation/operator.go","file.line":348},"message":"operation 'operation-install' skipped for fleet-server.8.4.1","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:47.075Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:11:47Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:47.154Z","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-13T14:11:47.164Z","log.origin":{"file.name":"process/stdlogger.go","file.line":54},"message":"fleet-server stderr: \"{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-09-13T14:11:47Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n{\\\"level\\\":\\\"info\\\",\\\"time\\\":\\\"2022-09-13T14:11:47Z\\\",\\\"message\\\":\\\"No applicable limit for 0 agents, using default.\\\"}\\n\"","agent.console.name":"fleet-server","agent.console.type":"stderr","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:47.958Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:48.757Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:11:48Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Waiting on active enrollment keys to be created in default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:51.961Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on active enrollment keys to be created in default policy with Fleet Server integration","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:11:59.770Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"app fleet-server--8.4.1-3f067576: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:59.770Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:11:59Z - message: Application: fleet-server--8.4.1[]: State changed to DEGRADED: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:11:59.963Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":773},"message":"Fleet Server - Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:00.297Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":471},"message":"Starting enrollment to URL: https://fleet-server-test-agent-http.elastic.svc:8220/","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:10.783Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:10.783Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:12:10Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Waiting on active enrollment keys to be created in default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:12:38.317Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"app fleet-server--8.4.1-3f067576: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:38.317Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:12:38Z - message: Application: fleet-server--8.4.1[]: State changed to DEGRADED: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:43.824Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:12:43.824Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:12:43Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Waiting on active enrollment keys to be created in default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:13:11.359Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"app fleet-server--8.4.1-3f067576: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:11.359Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:13:11Z - message: Application: fleet-server--8.4.1[]: State changed to DEGRADED: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:16.865Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:16.865Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:13:16Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Waiting on active enrollment keys to be created in default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:13:22.372Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"app fleet-server--8.4.1-3f067576: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:22.372Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:13:22Z - message: Application: fleet-server--8.4.1[]: State changed to DEGRADED: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:27.877Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:13:27.877Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:13:27Z - message: Application: fleet-server--8.4.1[]: State changed to STARTING: Waiting on active enrollment keys to be created in default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2022-09-13T14:14:00.915Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"degraded\": \"app fleet-server--8.4.1-3f067576: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process)\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:00.916Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:14:00Z - message: Application: fleet-server--8.4.1[]: State changed to DEGRADED: Running on default policy with Fleet Server integration; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:10.822Z","log.origin":{"file.name":"cmd/run.go","file.line":254},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:10.822Z","log.origin":{"file.name":"operation/operator.go","file.line":263},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:10.822Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: fleet-server--8.4.1","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-09-13T14:14:11.323Z","log.origin":{"file.name":"process/app.go","file.line":290},"message":"failed to stop fleet-server: os: process already finished","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:12.324Z","log.origin":{"file.name":"status/reporter.go","file.line":260},"message":"Elastic Agent status changed to \"online\": \"\"","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:12.324Z","log.origin":{"file.name":"cmd/run.go","file.line":262},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:12.324Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-09-13T14:14:12Z - message: Application: fleet-server--8.4.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-09-13T14:14:12.324Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":72},"message":"Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp 10.101.59.143:8220: connect: connection timed out
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.4/fleet-troubleshooting.html
I've couldn't find anything that is similar to the problem I face. Everyone seem to pass at least the Kibana UI Agents page, that shows the connected agents. On my end, I can see no agents configured, and it tells me I need to configure the fleet server first. On the other hand, I'm able to see agents are connected to the policy Elastic Agent on ECK policy...
I can't figure out from the logs what is the exact problem it is facing. Looks like the fleet-server is the problematic resource, and the agent is not running since it's not able to connect to the server, but the server logs doesn't have enough information for me to debug it.
Thanks in advance!