Thanks @MichelLaterman for getting back again.
Here are the logs (error and debug) for the Remote Fleet server where agents are not able to enroll to. Please note that the Real IP is changed to XX.XX.XX.XX in the below logs.
[elastic_agent][error] Checkin request to fleet-server succeeded after 125 failures
11:34:48.955
elastic_agent
[elastic_agent][error] checkin retry loop was stopped
11:35:25.381
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]
11:35:25.596
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:35:25.597
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:36:37.073
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
11:39:46.882
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
11:44:22.678
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:50:35.393
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
11:57:28.697
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
12:11:17.329
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]
12:16:35.989
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 1 failures
12:23:15.201
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 1 failures
14:38:13.086
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
14:43:20.317
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
14:57:11.478
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 4 failures
17:13:17.215
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:20:37.555
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:29:23.939
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:38:00.709
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:50:24.945
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 6 failures
17:54:24.532
elastic_agent
[elastic_agent][debug] running checks using 'DBus' controller
17:54:24.533
elastic_agent
[elastic_agent][debug] Error checker started
17:54:24.533
elastic_agent
[elastic_agent][debug] Crash checker started
17:54:24.533
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:25.397
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: 2be7dc2e-cf55-4bc3-826a-a46852b3d2ec, type: UPGRADE]
17:54:27.223
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:16:1, type: POLICY_CHANGE]
17:54:34.535
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:34.535
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:44.537
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:44.537
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:54.538
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:54.538
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:04.539
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:04.539
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:14.541
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:14.541
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:24.542
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:24.542
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:34.543
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:34.543
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:44.544
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:44.544
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:54.546
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:54.546
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:04.547
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:04.547
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:14.548
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:14.548
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:24.550
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:24.550
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:34.551
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:34.551
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:44.552
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:44.552
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:54.554
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:54.554
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:04.555
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:04.555
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:14.556
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:14.556
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:24.558
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:24.558
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:34.559
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:34.559
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:44.561
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:44.561
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:54.562
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:54.563
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:04.564
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:04.564
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:14.565
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:14.565
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:24.566
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:24.566
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:34.568
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:34.568
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:44.569
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:44.569
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:54.571
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:54.571
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:04.572
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:04.573
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:14.574
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:14.574
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:24.575
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:24.576
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:34.577
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:34.577
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:44.578
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:44.578
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:54.579
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:54.580
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:04.581
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:04.581
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:14.582
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:14.582
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:24.584
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:24.584
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:34.585
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:34.585
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:44.587
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:44.587
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:54.589
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:54.589
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:04.591
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:04.591
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:14.592
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:14.592
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:24.593
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:24.593
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:34.594
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:34.594
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:44.595
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:44.595
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:54.597
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:54.597
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:04.598
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:04.598
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:14.599
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:14.599
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:24.600
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:24.600
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:34.601
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:34.602
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:44.603
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:44.603
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:54.605
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:54.605
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:04.609
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:04.609
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:14.610
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:14.610
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:24.612
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:24.612
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:34.613
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:34.613
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:44.614
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:44.614
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:54.617
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:54.617
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:04.618
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:04:04.618
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:14.619
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:04:14.619
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:22.216
elastic_agent
[elastic_agent][debug] Cleaning up upgrade
18:04:24.216
elastic_agent
[elastic_agent][debug] Removing marker file
18:04:24.220
elastic_agent
[elastic_agent][debug] Removing previous symlink path
18:04:24.220
elastic_agent
[elastic_agent][debug] Removing hashed data directory
May 30, 2023
09:33:21.562
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
09:37:54.060
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
09:51:09.912
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 4 failures
May 31, 2023
11:09:14.793
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
11:09:18.909
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:09:19.180
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:09:19.180
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:10:47.797
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:13:42.038
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:17:34.219
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:23:22.735
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:28:39.817
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
14:57:09.277
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
14:57:11.685
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
14:58:58.066
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
15:04:52.788
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:43696->127.0.0.1:8221: read: connection reset by peer
15:11:09.623
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:38122->127.0.0.1:8221: read: connection reset by peer
15:17:24.416
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:60022->127.0.0.1:8221: read: connection reset by peer
15:25:38.163
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:34768->127.0.0.1:8221: read: connection reset by peer
15:27:51.618
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
15:30:04.429
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
15:33:59.953
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:58666->127.0.0.1:8221: read: connection reset by peer
15:38:26.191
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:51102->127.0.0.1:8221: read: connection reset by peer
15:46:11.083
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:37512->127.0.0.1:8221: read: connection reset by peer
15:53:55.995
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp [::1]:50792->[::1]:8221: read: connection reset by peer
16:02:40.284
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:48490->127.0.0.1:8221: read: connection reset by peer
Jun 1, 2023
09:42:14.832
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
09:42:18.021
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
09:42:18.467
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
09:42:18.467
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
09:43:32.686
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
09:47:01.135
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
09:50:32.487
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
Jun 2, 2023
10:51:50.252
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
11:02:55.472
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:02:55.780
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:02:55.780
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:03:57.472
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:06:27.012
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused
11:10:44.821
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused
11:14:50.233
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:57:35.334
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:47:32.828
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 273 failures
Showing entries until Jun 9, 11:47:32
Hope this helps.
Regards,