Yes, all servers are accessible.
I have more than 500 servers on which elastic-agent is installed and managed by fleet UI.
these servers disconnect 1 by 1 and after 2 days I only have a hundred or so servers connected: I lose my fleet check from which the status is offline except for the metrics and agent intergrations logs, which are still functional.
when I look at the logs of an offline server on the elastic-agent side, I get this:
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
11:10:54.312
elastic_agent
[elastic_agent][info] Starting server
11:10:54.312
elastic_agent
[elastic_agent][info] Starting stats endpoint
[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
11:10:54.324
elastic_agent
[elastic_agent][info] Updating running component model
11:10:54.475
elastic_agent
[elastic_agent][info] Spawned new component log-bb67d390-856c-11ee-814c-8124d3bd2dd3: Starting: spawned pid '1946051'
11:10:54.475
elastic_agent
[elastic_agent][info] Spawned new unit log-bb67d390-856c-11ee-814c-8124d3bd2dd3-logfile-system-7e9d3183-2cad-465b-ae62-0105c748c70c: Starting: spawned pid '1946051'
11:10:54.475
elastic_agent
[elastic_agent][info] Spawned new unit log-bb67d390-856c-11ee-814c-8124d3bd2dd3: Starting: spawned pid '1946051'
11:10:54.626
elastic_agent
[elastic_agent][info] Spawned new component system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3: Starting: spawned pid '1946063'
11:10:54.627
elastic_agent
[elastic_agent][info] Spawned new unit system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3: Starting: spawned pid '1946063'
11:10:54.627
elastic_agent
[elastic_agent][info] Spawned new unit system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3-system/metrics-system-7e9d3183-2cad-465b-ae62-0105c748c70c: Starting: spawned pid '1946063'
11:10:54.830
elastic_agent
[elastic_agent][info] component model updated
11:10:54.830
elastic_agent
[elastic_agent][info] Updating running component model
11:10:54.853
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
11:10:54.853
elastic_agent
[elastic_agent][info] Component state changed system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3 (STARTING->HEALTHY): Healthy: communicating with pid '1946063'
11:10:54.887
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
11:10:54.887
elastic_agent
[elastic_agent][info] Component state changed log-bb67d390-856c-11ee-814c-8124d3bd2dd3 (STARTING->HEALTHY): Healthy: communicating with pid '1946051'
11:10:55.041
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
11:10:55.149
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
11:10:55.261
elastic_agent
[elastic_agent][info] control checkin v2 protocol has chunking enabled
11:10:55.274
elastic_agent
[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
11:10:55.877
elastic_agent
[elastic_agent][info] Unit state changed system/metrics-******0-856c-11ee-814c-8124d3bd2dd3-system/metrics-system-7e9d3183-2cad-465b-ae62-0105c748c70c (STARTING->HEALTHY): Healthy
11:10:55.877
elastic_agent
[elastic_agent][info] Unit state changed system/metrics-b******-856c-11ee-814c-8124d3bd2dd3 (STARTING->HEALTHY): Healthy
11:10:55.892
elastic_agent
[elastic_agent][info] Unit state changed log-bb67d390-856c-11ee-814c-8124d3bd2dd3 (STARTING->HEALTHY): Healthy
11:10:55.893
elastic_agent
[elastic_agent][info] Unit state changed log-bb67d390-856c-11ee-814c-8124d3bd2dd3-logfile-system-*******2cad-465b-ae62-0105c748c70c (STARTING->HEALTHY): Healthy
11:12:48.695
elastic_agent
[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
11:16:20.038
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:20:47.409
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:30:31.553
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:38:16.216
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
when I connect to a server managed by fleet and recently disconnected to check the elastic agent status :
elastic-agent status --output json
{
"info": {
"id": "340ec97f-bb76-4e19-bcf5-ff37dce91f62",
"version": "8.14.2",
"commit": "1738179d53e747c48af7350a0b8fe68eda1a5b31",
"build_time": "2024-07-01 16:29:34 +0000 UTC",
"snapshot": false,
"pid": 1946034,
"unprivileged": false
},
"state": 2,
"message": "Running",
"components": [
{
"id": "beat/metrics-monitoring",
"name": "beat/metrics",
"state": 2,
"message": "Healthy: communicating with pid '1946082'",
"units": [
{
"unit_id": "beat/metrics-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "beat/metrics-monitoring-metrics-monitoring-beats",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"meta": {
"build_time": "2024-07-01 16:59:44 +0000 UTC",
"commit": "e9455e203842edf9086f34b3ca2fa2b08bc76081"
}
}
},
{
"id": "filestream-monitoring",
"name": "filestream",
"state": 2,
"message": "Healthy: communicating with pid '1946071'",
"units": [
{
"unit_id": "filestream-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "filestream-monitoring-filestream-monitoring-agent",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"meta": {
"build_time": "2024-07-01 16:59:44 +0000 UTC",
"commit": "e9455e203842edf9086f34b3ca2fa2b08bc76081"
}
}
},
{
"id": "http/metrics-monitoring",
"name": "http/metrics",
"state": 2,
"message": "Healthy: communicating with pid '1946090'",
"units": [
{
"unit_id": "http/metrics-monitoring",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "http/metrics-monitoring-metrics-monitoring-agent",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"meta": {
"build_time": "2024-07-01 16:59:44 +0000 UTC",
"commit": "e9455e203842edf9086f34b3ca2fa2b08bc76081"
}
}
},
{
"id": "log-bb67d390-856c-11ee-814c-8124d3bd2dd3",
"name": "log",
"state": 2,
"message": "Healthy: communicating with pid '1946051'",
"units": [
{
"unit_id": "log-bb67d390-856c-11ee-814c-8124d3bd2dd3",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "log-bb67d390-856c-11ee-814c-8124d3bd2dd3-logfile-system-7e9d3183-2cad-465b-ae62-0105c748c70c",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"meta": {
"build_time": "2024-07-01 16:59:44 +0000 UTC",
"commit": "e9455e203842edf9086f34b3ca2fa2b08bc76081"
}
}
},
{
"id": "system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3",
"name": "system/metrics",
"state": 2,
"message": "Healthy: communicating with pid '1946063'",
"units": [
{
"unit_id": "system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3",
"unit_type": 1,
"state": 2,
"message": "Healthy"
},
{
"unit_id": "system/metrics-bb67d390-856c-11ee-814c-8124d3bd2dd3-system/metrics-system-7e9d3183-2cad-465b-ae62-0105c748c70c",
"unit_type": 0,
"state": 2,
"message": "Healthy"
}
],
"version_info": {
"name": "beat-v2-client",
"meta": {
"build_time": "2024-07-01 16:59:44 +0000 UTC",
"commit": "e9455e203842edf9086f34b3ca2fa2b08bc76081"
}
}
}
],
"FleetState": 4,
"FleetMessage": "status code: 500, fleet-server returned an error: BadRequest, message: failed to update upgrade_details: upgrade_details no action for id \"\" found",
"upgrade_details": {
"target_version": "8.14.2",
"state": "UPG_ROLLBACK",
"metadata": {}
}
}
