Hi, I've got a problem with installing fleet server.
Fleet server enrolls with no errors, but kibana it's stuck in "updating" state.
I was trying to install it with my own ssl (in pem format), with quick start (insecure), insecure with default URL (localhost:8220) - all of them result in "updating" state.
Logs point a problem with connection to 8220 port.
Tried both RPM and TAR elastic-agent clients.
My configuration is:
ELK 8.0
Public network, ports 5601, 9200, 8220 open.
Hosts record with dns to 127.0.0.1 for local communication.
SSL (production, not self signed) for ES/Kibana.
KIBANA, ES, Fleet Server - on the same self-hosted vm.
Showing entries from Mar 1, 13:55:44
13:55:44.914
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:44.914
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:44.915
elastic_agent
[elastic_agent][info] Agent is managed locally
13:55:44.915
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
13:55:44.915
elastic_agent
[elastic_agent][info] Agent is managed locally
13:55:44.915
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
13:55:46.189
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
13:55:46.189
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
13:55:46.193
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:46.193
elastic_agent
[elastic_agent][info] Agent is starting
13:55:46.193
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:46.193
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:46.193
elastic_agent
[elastic_agent][info] Agent is starting
13:55:46.193
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:46.195
elastic_agent
[elastic_agent][info] Agent is stopped
13:55:46.195
elastic_agent
[elastic_agent][info] Agent is stopped
13:55:46.196
elastic_agent
[elastic_agent][info] Configuration changes detected
13:55:46.196
elastic_agent
[elastic_agent][info] Configuration changes detected
13:55:46.434
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:46.434
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:46.436
elastic_agent
[elastic_agent][info] Agent is in Fleet Server bootstrap mode
13:55:46.436
elastic_agent
[elastic_agent][info] Agent is in Fleet Server bootstrap mode
13:55:46.891
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:46.891
elastic_agent
[elastic_agent][info] Agent is starting
13:55:46.891
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:46.891
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:46.891
elastic_agent
[elastic_agent][info] Agent is starting
13:55:46.891
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:46.893
elastic_agent
[elastic_agent][info] Agent is stopped
13:55:46.893
elastic_agent
[elastic_agent][info] Agent is stopped
13:55:46.900
elastic_agent
[elastic_agent][info] New State ID is bW8QdZ3y
13:55:46.900
elastic_agent
[elastic_agent][info] Converging state requires execution of 1 step(s)
13:55:46.900
elastic_agent
[elastic_agent][info] New State ID is bW8QdZ3y
13:55:46.900
elastic_agent
[elastic_agent][info] Converging state requires execution of 1 step(s)
13:55:48.718
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:48+01:00 - message: Application: fleet-server--8.0.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:55:48.718
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:48+01:00 - message: Application: fleet-server--8.0.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:55:48.719
elastic_agent
[elastic_agent][info] Updating internal state
13:55:48.719
elastic_agent
[elastic_agent][info] Updating internal state
13:55:50.295
elastic_agent
[elastic_agent][warn] Elastic Agent status changed to: 'degraded'
13:55:50.295
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:50+01:00 - message: Application: fleet-server--8.0.0[]: State changed to DEGRADED: Running on policy with Fleet Server integration: 499b5aa7-d214-5b5d-838b-3cd76469844e; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'
13:55:50.295
elastic_agent
[elastic_agent][warn] Elastic Agent status changed to: 'degraded'
13:55:50.295
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:50+01:00 - message: Application: fleet-server--8.0.0[]: State changed to DEGRADED: Running on policy with Fleet Server integration: 499b5aa7-d214-5b5d-838b-3cd76469844e; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'
13:55:54.744
elastic_agent
[elastic_agent][info] waiting for installer of pipeline 'default' to finish
13:55:54.744
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: fleet-server--8.0.0
13:55:54.744
elastic_agent
[elastic_agent][info] waiting for installer of pipeline 'default' to finish
13:55:54.744
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: fleet-server--8.0.0
13:55:56.245
elastic_agent
[elastic_agent][info] Elastic Agent status changed to: 'online'
13:55:56.245
elastic_agent
[elastic_agent][info] Elastic Agent status changed to: 'online'
13:55:56.246
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:56+01:00 - message: Application: fleet-server--8.0.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
13:55:56.246
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:56+01:00 - message: Application: fleet-server--8.0.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
13:55:56.478
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:56.478
elastic_agent
[elastic_agent][info] Detecting execution mode
13:55:56.481
elastic_agent
[elastic_agent][info] Agent is managed by Fleet
13:55:56.481
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
13:55:56.481
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:55:56.481
elastic_agent
[elastic_agent][info] Agent is managed by Fleet
13:55:56.481
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
13:55:56.481
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
13:55:57.285
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
13:55:57.285
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
13:55:57.288
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:57.288
elastic_agent
[elastic_agent][info] Agent is starting
13:55:57.288
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:57.288
elastic_agent
[elastic_agent][info] Starting stats endpoint
13:55:57.288
elastic_agent
[elastic_agent][info] Agent is starting
13:55:57.288
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
13:55:57.401
elastic_agent
[elastic_agent][info] New State ID is Eqj-AzeP
13:55:57.401
elastic_agent
[elastic_agent][info] Converging state requires execution of 2 step(s)
13:55:57.401
elastic_agent
[elastic_agent][info] New State ID is Eqj-AzeP
13:55:57.401
elastic_agent
[elastic_agent][info] Converging state requires execution of 2 step(s)
13:55:57.511
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for fleet-server.8.0.0
13:55:57.511
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for fleet-server.8.0.0
13:55:58.085
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:58+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:55:58.085
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:58+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:55:59.153
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:59+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running on policy with Fleet Server integration: 499b5aa7-d214-5b5d-838b-3cd76469844e - type: 'STATE' - sub_type: 'RUNNING'
13:55:59.153
elastic_agent
[elastic_agent][info] 2022-03-01T13:55:59+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running on policy with Fleet Server integration: 499b5aa7-d214-5b5d-838b-3cd76469844e - type: 'STATE' - sub_type: 'RUNNING'
13:56:06.090
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:06+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:06.090
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:06+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:10.392
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:10+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:56:10.392
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:10+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:56:12.845
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:12+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:12.845
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:12+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:12.850
elastic_agent
[elastic_agent][info] Updating internal state
13:56:12.850
elastic_agent
[elastic_agent][info] Updating internal state
13:56:14.261
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:14+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:56:14.261
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:14+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:56:16.137
elastic_agent
[elastic_agent][info] New State ID is U1GyoXz6
13:56:16.137
elastic_agent
[elastic_agent][info] Converging state requires execution of 2 step(s)
13:56:16.241
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for fleet-server.8.0.0
13:56:16.241
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for fleet-server.8.0.0
13:56:16.637
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.0.0
13:56:16.638
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for filebeat.8.0.0
13:56:16.640
elastic_agent
[elastic_agent][info] initiating restart of 'filebeat_monitoring' due to config change
13:56:16.676
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:16+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STOPPING: Stopping - type: 'STATE' - sub_type: 'STOPPING'
13:56:16.812
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:16+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RESTARTING: exited with code: 0 - type: 'STATE' - sub_type: 'STARTING'
13:56:16.812
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:16+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:16.812
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:16+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
13:56:17.385
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:17+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:18.154
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:18+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
13:56:18.733
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:18+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:19.212
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for metricbeat.8.0.0
13:56:19.212
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for metricbeat.8.0.0
13:56:19.214
elastic_agent
[elastic_agent][info] initiating restart of 'metricbeat_monitoring' due to config change
13:56:19.728
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:19+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
13:56:20.363
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:20+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
13:56:20.363
elastic_agent
[elastic_agent][info] Updating internal state
13:56:20.373
elastic_agent
[elastic_agent][error] failed to dispatch actions, error: acknowledge 1 actions '[action_id: policy:499b5aa7-d214-5b5d-838b-3cd76469844e:9:1, type: POLICY_CHANGE]' for elastic-agent '7551366c-c453-45ef-8bcb-b750ed23ff4a' failed: fail to ack to fleet: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/acks?": dial tcp [::1]:8220: connect: connection refused
13:56:20.374
elastic_agent
[elastic_agent][error] Elastic Agent status changed to: 'error'
13:56:21.538
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp 127.0.0.1:8220: connect: connection refused
13:56:23.003
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:23+01:00 - message: Application: filebeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:56:24.868
elastic_agent
[elastic_agent][info] 2022-03-01T13:56:24+01:00 - message: Application: metricbeat--8.0.0--36643631373035623733363936343635[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
13:57:22.190
elastic_agent
[elastic_agent][info] 2022-03-01T13:57:22+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
13:57:33.244
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp 127.0.0.1:8220: connect: connection refused
13:58:22.196
elastic_agent
[elastic_agent][error] 2022-03-01T13:58:22+01:00 - message: Application: fleet-server--8.0.0[7551366c-c453-45ef-8bcb-b750ed23ff4a]: State changed to FAILED: Missed two check-ins - type: 'ERROR' - sub_type: 'FAILED'
14:01:33.176
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp 127.0.0.1:8220: connect: connection refused
14:07:11.326
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp 127.0.0.1:8220: connect: connection refused
14:13:34.094
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp [::1]:8220: connect: connection refused
14:22:03.384
elastic_agent
[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "http://localhost:8220/api/fleet/agents/7551366c-c453-45ef-8bcb-b750ed23ff4a/checkin?": dial tcp 127.0.0.1:8220: connect: connection refused