Fail to checkin to fleet-server

ethical20 · May 24, 2023, 11:31am

Hi All,

I have successfully enrolled my remote server/machine into my Fleet server and I can see my metrics and logs coming thru.

The issue is that at the beginning of the enrollment the status of the agent in kibana was Updating then it turned to Offline without ever being Online till now.

The fleet server is Online:

Here are the results of elastic agent status:


elastic-agent status
State: HEALTHY
Message: Running
Fleet State: FAILED
Fleet Message: fail to checkin to fleet-server: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/f82222be-ec5e-49e2-a584-4f9c74bcf610/checkin?": dial tcp [::1]:8221: connect: network is unreachable


Components:
  * filestream      (HEALTHY)
                    Healthy: communicating with pid '795'
  * log             (HEALTHY)
                    Healthy: communicating with pid '775'
  * system/metrics  (HEALTHY)
                    Healthy: communicating with pid '780'
  * beat/metrics    (HEALTHY)
                    Healthy: communicating with pid '785'
  * http/metrics    (HEALTHY)
                    Healthy: communicating with pid '786'

The error showed in the elastic agent log file opt/Elastic/Agent/data/elastic-agent-10dc6a/logs/elastic-agent-20230524.ndjson is this:


{"log.level":"error","@timestamp":"2023-05-24T09:34:18.986Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":197},"message":"Cannot checkin in with fleet-server, retrying","log":{"source":"elastic-agent"},"error":{"message":"fail to checkin to fleet-server: all hosts failed: 1 error occurred:\n\t* requester 0/1 to host https://localhost:8221/ errored: Post \"https://localhost:8221/api/fleet/agents/f82222be-ec5e-49e2-a584-..../checkin?\": dial tcp [::1]:8221: connect: network is unreachable\n\n"},"request_duration_ns":237691,"failed_checkins":100,"retry_after_ns"::553251097583,"ecs.version":"1.6.0"}

Why is the agent is trying to connect to port 8221 ? In the enrollment it was 8220 !

I've passed by this thread but couldn't understand or find a solution.

github.com/elastic/elastic-agent

Change local fleet-server connection to localhost:8221

elastic:main ← michel-laterman:fleet-port

opened 09:47PM - 01 Dec 22 UTC

michel-laterman

+41 -0

## What does this PR do? Fix an issue where the local fleet-server port was n…ot properly used by the elastic-agent when running an instance of fleet-server. ## Why is it important? When running a large scale of agents, the fleet-server may hit the connection limit if it's running on the same port as the other agents. this causes the elastic-agent running the fleet-server to be marked as degraded. ## Checklist - [x] My code follows the style guidelines of this project - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added an entry in `CHANGELOG.next.asciidoc` or `CHANGELOG-developer.next.asciidoc`.

Your help is much appreciated.

ethical20 · May 25, 2023, 7:28am

Dears @ stephenb @ axw @ leandrojmp

I can see that you are the champions of the Elastic Agent support forum, actually I searched a lot in this issue with no one ever mentioned about this before.

Can you please take a look into this?

Thanks in advance and sorry for any annoyance.

Cristina_Amico · May 25, 2023, 1:55pm

Hi @ethical20,

Could you provide the command used for the enrollment and a screenshot of the Fleet > Settings tab?

Thanks,
Cristina

ethical20 · May 25, 2023, 2:10pm

Hi @Cristina_Amico

Here is the command used for enrollment:

./elastic-agent install --url=https://IP:5045 --fleet-server-es=https://IP:5050 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2ODQ4NTUwODY5NjM6ZDMtd3VybGJTNEdXWE1.... --enrollment-token=QktyWlVvZ0I0TC16Q05jdFU0ZzM6d2VaYUtLbndTeDZpeWhTS....== --certificate-authorities=/etc/ssl/certs/elasticsearch-ca.pem

And here is the Fleet Setting Page:

Thanks in advance

Cristina_Amico · May 25, 2023, 2:35pm

In the provided command to enroll the agent I see that there is an option --fleet-server-service-token that shouldn't be there, it should be used only when enrolling a fleet server.

./elastic-agent install \
  --url=https://IP:5045 \
  --fleet-server-es=https://IP:5050 \
  --fleet-server-service-token=*** \
  --enrollment-token=*** \
  --certificate-authorities=/etc/ssl/certs/elasticsearch-ca.pem

There is a reference of the available commands at this linkhere. Could you try enrolling the agent without this flag and see how it goes?

Thanks,
Cristina

ethical20 · May 26, 2023, 12:35pm

Thanks @Cristina_Amico

I've first tried your solution and removed the --fleet-server-service-token and I got this error:

{"log.level":"info","@timestamp":"2023-05-26T11:15:34.180Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":407},"message":"Generating self-signed certificate for Fleet Server","ecs.version":"1.6.0"}
Error: invalid connection string: must include a service token
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.7/fleet-troubleshooting.html
Error: enroll command failed with exit code: 1

I then removed the --fleet-server-es

./elastic-agent install --url=https://IP:5045 --enrollment-token=*** --certificate-authorities=/etc/ssl/certs/elasticsearch-ca.pem
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2023-05-26T11:16:40.287Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":475},"message":"Starting enrollment to URL: https://IP:5045/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp IP:5045: connect: connection refused

I've checked the server having the connection refused and it has the following message:

connect_to 192.68.0.7 port 8220: failed

I've checked my fleet server and I can see following port configs:

kibana@kibana:~$ netstat -tulpn
kibana@kibana:~$ netstat -tulpn
(Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8221          0.0.0.0:*               LISTEN      708/fleet-server
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      669/sshd: /usr/sbin
tcp        0      0 192.168.0.7:5601        0.0.0.0:*               LISTEN      611/node
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      590/systemd-resolve
tcp        0      0 127.0.0.1:6791          0.0.0.0:*               LISTEN      607/elastic-agent
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      607/elastic-agent
tcp6       0      0 :::22                   :::*                    LISTEN      669/sshd: /usr/sbin
tcp6       0      0 :::8220                 :::*                    LISTEN      708/fleet-server
udp        0      0 127.0.0.53:53           0.0.0.0:*                           590/systemd-resolve
udp6       0      0 fe80::5c6b:3cff:... :::*                                588/systemd-network

kibana@kibana:~$ sudo netstat -pln | grep 8220
tcp6       0      0 :::8220                 :::*                    LISTEN      708/fleet-server

Does this mean Fleet server is listening on 8220 only for IPV6? Although Im only using IPV4.

Any clue in making fleet server listen to 8220 ?

Thanks

MichelLaterman · May 30, 2023, 6:11pm

Are you intending to run a local fleet-server process on the agent?

From the context in this thread I don't think you are, so the --fleet-server-* flags should be removed from the install command (as you did).

In your comment, you try to enroll to IP:5045, is that where fleet-server is running?
Can you run curl -v IP:5045/api/status to verify if the server is running?

How did you bootstrap the fleet-server?
One stumbling block we have is that in order to run on a non-default port (5045 instead of 8220) you need to pass the --fleet-server-port=5045 flag on installation (the --url option is not used to specify the port on fleet-server installation)

Finally, what is the fleet-server host in Kibana?

ethical20 · June 1, 2023, 10:30am

Thanks @MichelLaterman

Please be patient about the below detailed scenario.

The kibana host you see in the below image is where I've already installed Kibana service on this local machine https://192.168.0.7:5601 plus I've installed a Fleet server on the same machine which should be listening on 192.168.0.7:8220

Also I have another Fleet server which is remote (not local)

Both fleet servers now are Online and Healthy .

Enrolling a local machine 192.168.0.101 to Local Fleet server 192.168.0.7 at port 8220 succussed.

My Issue now is with agent enrollment for a remote machine on remote fleet server.

To get out of the non default ports issue 5045 that might mislead us, my infrastructure now has the below 2 Fleet servers, (local and remote.) both on 8220

Trying to enroll a remote machine to remote fleet server gave the below error:

~/elastic-agent-8.8.0-linux-x86_64# ./elastic-agent install --url=https://IP:8220 --enrollment-token=*** --certificate-authorities=/etc/ssl/certs/elasticsearch-ca.pem
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2023-06-01T08:25:11.177Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: https://IP:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp IP:8220: connect: connection timed out
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.8/fleet-troubleshooting.html

curl -v IP:8200/api/status result is :

curl -v https://IP:8200/api/status
*   Trying IP:8200...
* connect to IP port 8200 failed: Connection timed out
* Failed to connect to IP port 8200 after 129499 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to IP port 8200 after 129499 ms: Connection timed out

Surprisingly curl -v https://192.16.0.7:8200/api/status which is online and Healthy and I can enroll agent successfully gave this

*   Trying 192.16.0.7:8200...
* connect to 192.16.0.7 port 8200 failed: Connection timed out
* Failed to connect to 192.16.0.7 port 8200 after 129895 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 192.16.0.7 port 8200 after 129895 ms: Connection timed out

The only way to get a machine enrolled to the Fleet is by installing it as a Fleet server using this command:

./elastic-agent install --url=https://IP:8220 \
  --fleet-server-es=https://IP:5050 \
  --fleet-server-service-token=AAEAAWV*** \
  --fleet-server-policy=fleet-server-policy \
  --certificate-authorities=/root/certs/elasticsearch-ca.pem \
  --fleet-server-es-ca=/root/certs/elasticsearch-ca.pem \
  --fleet-server-cert=/root/certs/fleet-server.crt \
  --fleet-server-cert-key=/root/certs/fleet-server.key

This is the only way I can enroll agents in Fleet, but they will have the role of a Fleet server instead of an agent.

Any help is much appreciated.

MichelLaterman · June 1, 2023, 4:09pm

Are you using the argument --url=https://IP:8220 verbatim, or --url=https://192.168.0.7:8220

ethical20 · June 2, 2023, 8:19am

Hi @MichelLaterman

I'm using the argument --url=https://XX.XX.XX.XX:8220 where XX.XX.XX.XX or what I'm referring to IP is the IP of my remote fleet server, The --url=https://192.168.0.7:8220 I'm using it for my local machines to connect to it as a local Fleet server.

In other words:

--url=https://RemoteIP:8220 is the Remote fleet server

and
--url=https://192.168.0.7:8220 is my local fleet server

Hope this help, thanks in advance

MichelLaterman · June 5, 2023, 8:55pm

you're listing port 8220 on Kibana and using port 8200 in curl.

MichelLaterman · June 5, 2023, 9:24pm

ethical20:

Trying to enroll a remote machine to remote fleet server gave the below error:

~/elastic-agent-8.8.0-linux-x86_64# ./elastic-agent install --url=https://IP:8220 --enrollment-token=*** --certificate-authorities=/etc/ssl/certs/elasticsearch-ca.pem
Elastic Agent will be installed at /opt/Elastic/Agent and will run as a service. Do you want to continue? [Y/n]:y
{"log.level":"info","@timestamp":"2023-06-01T08:25:11.177Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":478},"message":"Starting enrollment to URL: https://IP:8220/","ecs.version":"1.6.0"}
Error: fail to enroll: fail to execute request to fleet-server: dial tcp IP:8220: connect: connection timed out
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.8/fleet-troubleshooting.html

for this issue, i would check if curl with 8220 works as expected as this looks like a network issue

ethical20 · June 6, 2023, 7:36am

Thanks @MichelLaterman for getting back.

From this netstat can you see if there is an issue at some point? cause from my side I see everything is normal!

kibana@kibana:~$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      597/systemd-resolve
tcp        0      0 127.0.0.1:8221          0.0.0.0:*               LISTEN      1096/fleet-server
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      675/sshd: /usr/sbin
tcp        0      0 127.0.0.1:6791          0.0.0.0:*               LISTEN      614/elastic-agent
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      614/elastic-agent
tcp        0      0 192.168.0.7:5601        0.0.0.0:*               LISTEN      618/node
tcp6       0      0 :::22                   :::*                    LISTEN      675/sshd: /usr/sbin
tcp6       0      0 :::8220                 :::*                    LISTEN      1096/fleet-server
udp        0      0 127.0.0.53:53           0.0.0.0:*                           597/systemd-resolve
udp6       0      0 fe80::5c6b:3cff:fe6:546 :::*                                595/systemd-network

Trying to curl from the Elasticsearch machine to the fleet server machine

elk@elk:~$ curl -v https://192.16.0.7:8200/api/status
*   Trying 192.16.0.7:8200...
* connect to 192.16.0.7 port 8200 failed: Connection timed out
* Failed to connect to 192.16.0.7 port 8200 after 129411 ms: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to 192.16.0.7 port 8200 after 129411 ms: Connection timed out

Back to fleet server...

kibana@kibana:~$ sudo netstat -pln | grep 8220
tcp6       0      0 :::8220                 :::*                    LISTEN      1096/fleet-server

Shouldn't this be as below, as I understand that fleet server is using ipv6 (and I don't know why!) Is there a config file in fleet server to force it to listen on IPV4 and disable IPV6?

tcp  0.0.0.0:8220

Thanks in advance.

MichelLaterman · June 6, 2023, 6:59pm

Once again you are using curl with port 8200 and the server should be on 8220.
We can see (as you said) the server is running in IPv6.

Can you provide the logs from the server? or a diagnostics bundle with elastic-agent diagnostics from the agent managing the fleet-server?

ethical20 · June 9, 2023, 10:56am

Thanks @MichelLaterman for getting back again.

Here are the logs (error and debug) for the Remote Fleet server where agents are not able to enroll to. Please note that the Real IP is changed to XX.XX.XX.XX in the below logs.

[elastic_agent][error] Checkin request to fleet-server succeeded after 125 failures
11:34:48.955
elastic_agent
[elastic_agent][error] checkin retry loop was stopped
11:35:25.381
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]
11:35:25.596
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:35:25.597
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:36:37.073
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

11:39:46.882
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

11:44:22.678
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:50:35.393
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

11:57:28.697
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

12:11:17.329
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:15:1, type: POLICY_CHANGE]
12:16:35.989
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 1 failures
12:23:15.201
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 1 failures
14:38:13.086
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
14:43:20.317
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
14:57:11.478
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 4 failures
17:13:17.215
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:20:37.555
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:29:23.939
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:38:00.709
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
17:50:24.945
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 6 failures
17:54:24.532
elastic_agent
[elastic_agent][debug] running checks using 'DBus' controller
17:54:24.533
elastic_agent
[elastic_agent][debug] Error checker started
17:54:24.533
elastic_agent
[elastic_agent][debug] Crash checker started
17:54:24.533
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:25.397
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: 2be7dc2e-cf55-4bc3-826a-a46852b3d2ec, type: UPGRADE]
17:54:27.223
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:16:1, type: POLICY_CHANGE]
17:54:34.535
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:34.535
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:44.537
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:44.537
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:54:54.538
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:54:54.538
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:04.539
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:04.539
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:14.541
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:14.541
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:24.542
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:24.542
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:34.543
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:34.543
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:44.544
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:44.544
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:55:54.546
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:55:54.546
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:04.547
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:04.547
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:14.548
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:14.548
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:24.550
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:24.550
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:34.551
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:34.551
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:44.552
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:44.552
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:56:54.554
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:56:54.554
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:04.555
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:04.555
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:14.556
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:14.556
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:24.558
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:24.558
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:34.559
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:34.559
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:44.561
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:44.561
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:57:54.562
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:57:54.563
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:04.564
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:04.564
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:14.565
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:14.565
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:24.566
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:24.566
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:34.568
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:34.568
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:44.569
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:44.569
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:58:54.571
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:58:54.571
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:04.572
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:04.573
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:14.574
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:14.574
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:24.575
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:24.576
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:34.577
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:34.577
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:44.578
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:44.578
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
17:59:54.579
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
17:59:54.580
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:04.581
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:04.581
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:14.582
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:14.582
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:24.584
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:24.584
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:34.585
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:34.585
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:44.587
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:44.587
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:00:54.589
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:00:54.589
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:04.591
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:04.591
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:14.592
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:14.592
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:24.593
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:24.593
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:34.594
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:34.594
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:44.595
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:44.595
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:01:54.597
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:01:54.597
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:04.598
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:04.598
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:14.599
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:14.599
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:24.600
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:24.600
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:34.601
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:34.602
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:44.603
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:44.603
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:02:54.605
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:02:54.605
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:04.609
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:04.609
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:14.610
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:14.610
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:24.612
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:24.612
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:34.613
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:34.613
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:44.614
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:44.614
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:03:54.617
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:03:54.617
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:04.618
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:04:04.618
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:14.619
elastic_agent
[elastic_agent][debug] retrieved service PID [418] changed 1 times within 6
18:04:14.619
elastic_agent
[elastic_agent][debug] watcher having PID: 3656
18:04:22.216
elastic_agent
[elastic_agent][debug] Cleaning up upgrade
18:04:24.216
elastic_agent
[elastic_agent][debug] Removing marker file
18:04:24.220
elastic_agent
[elastic_agent][debug] Removing previous symlink path
18:04:24.220
elastic_agent
[elastic_agent][debug] Removing hashed data directory
May 30, 2023
09:33:21.562
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
09:37:54.060
elastic_agent
[elastic_agent][error] Cannot checkin in with fleet-server, retrying
09:51:09.912
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 4 failures
May 31, 2023
11:09:14.793
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
11:09:18.909
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:09:19.180
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:09:19.180
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:10:47.797
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:13:42.038
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:17:34.219
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:23:22.735
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:28:39.817
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

14:57:09.277
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
14:57:11.685
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
14:58:58.066
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

15:04:52.788
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:43696->127.0.0.1:8221: read: connection reset by peer

15:11:09.623
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:38122->127.0.0.1:8221: read: connection reset by peer

15:17:24.416
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:60022->127.0.0.1:8221: read: connection reset by peer

15:25:38.163
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:34768->127.0.0.1:8221: read: connection reset by peer

15:27:51.618
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
15:30:04.429
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
15:33:59.953
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:58666->127.0.0.1:8221: read: connection reset by peer

15:38:26.191
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:51102->127.0.0.1:8221: read: connection reset by peer

15:46:11.083
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:37512->127.0.0.1:8221: read: connection reset by peer

15:53:55.995
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp [::1]:50792->[::1]:8221: read: connection reset by peer

16:02:40.284
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": read tcp 127.0.0.1:48490->127.0.0.1:8221: read: connection reset by peer

Jun 1, 2023
09:42:14.832
elastic_agent
[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
09:42:18.021
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
09:42:18.467
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
09:42:18.467
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
09:43:32.686
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

09:47:01.135
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

09:50:32.487
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
Jun 2, 2023
10:51:50.252
elastic_agent

[elastic_agent][debug] update marker not present at '/opt/Elastic/Agent/data'
11:02:55.472
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:02:55.780
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default-fleet-server-fleet_server-cf05565f-4dec-4d32-b8e4-714d573be492 (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:02:55.780
elastic_agent
[elastic_agent][error] Unit state changed fleet-server-default (STARTING->FAILED): Error - dial tcp XX.XX.XX.XX:5050: connect: connection refused
11:03:57.472
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:06:27.012
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp [::1]:8221: connect: connection refused

11:10:44.821
elastic_agent
[elastic_agent][error] ack retrier: commit failed with error: acknowledge 1 actions '[action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]' for elastic-agent 'aa84adda-aa24-40d7-8b50-fb698f5a1db5' failed: fail to ack to fleet: all hosts failed: 1 error occurred:
	* requester 0/1 to host https://localhost:8221/ errored: Post "https://localhost:8221/api/fleet/agents/aa84adda-aa24-40d7-8b50-fb698f5a1db5/acks?": dial tcp 127.0.0.1:8221: connect: connection refused

11:14:50.233
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
11:57:35.334
elastic_agent

[elastic_agent][error] Cannot checkin in with fleet-server, retrying
11:47:32.828
elastic_agent
[elastic_agent][error] Checkin request to fleet-server succeeded after 273 failures
Showing entries until Jun 9, 11:47:32

Hope this helps.

Regards,

MichelLaterman · June 9, 2023, 5:35pm

Thanks for the elastic-agent logs.
We can see from your earlier post that fleet-server is running (in IPv4) on 127.0.0.1:8221, and for some reason on IPv6 on :::8220
Some of these messages are very strange to me, we can see it attempt to connect to fleet-server on the local host on both IPv4 and IPv6:

dial tcp 127.0.0.1:8221: connect: connection refused
dial tcp [::1]:8221: connect: connection refused

I'm not sure why the IPv4 connection is being refused - the only thing I've seen that's similar is when port mapping for a container/vm is active,
for example:

$ lsof | grep 8220
VBoxHeadl  9632 laterman   12u     IPv4 0x63e83566b40cb98d          0t0                 TCP *:8220 (LISTEN)
fleet-ser 10291 laterman   14u     IPv6 0x63e8355d1ea6c525          0t0                 TCP *:8220 (LISTEN)

Can you confirm if there are any containers or vms mapping ports?

Can you also provide the logs from when fleet-server has started? There are some messages that contain the listening address.

ethical20 · June 12, 2023, 12:33pm

Thanks @MichelLaterman for the follow up.

What I've done now is that I have got a new server, installed fleet server on it and agents are now connecting normally to the new server.

But to be honest I couldn't identify the issue behind such thing.

Here are the results from the new working server:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:8221          0.0.0.0:*               LISTEN      424069/fleet-server 
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      423798/elastic-agen 
tcp        0      0 127.0.0.1:6791          0.0.0.0:*               LISTEN      423798/elastic-agen 
tcp        0      0 0.0.0.0:5355            0.0.0.0:*               LISTEN      251718/systemd-reso 
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      251718/systemd-reso 
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      250/sshd: /usr/sbin 
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      521/exim4           
tcp6       0      0 :::8220                 :::*                    LISTEN      424069/fleet-server 
tcp6       0      0 :::5355                 :::*                    LISTEN      251718/systemd-reso 
tcp6       0      0 :::22                   :::*                    LISTEN      250/sshd: /usr/sbin 
tcp6       0      0 ::1:25                  :::*                    LISTEN      521/exim4           
udp        0      0 127.0.0.53:53           0.0.0.0:*                           251718/systemd-reso 
udp        0      0 0.0.0.0:5355            0.0.0.0:*                           251718/systemd-reso 
udp6       0      0 :::5355                 :::*                                251718/systemd-reso 
root@newsrv:~# netstat -pln | grep 8220
tcp6       0      0 :::8220                 :::*                    LISTEN      424069/fleet-server

Curl from Agent to new server

url https://NN.NN.NN.NN:8220/api/status
curl: (60) SSL certificate problem: unable to get local issuer certificate

elastic-agent status
State: HEALTHY
Message: Running
Fleet State: HEALTHY
Fleet Message: Connected
Components:
  * log             (HEALTHY)
                    Healthy: communicating with pid '424058'
  * system/metrics  (HEALTHY)
                    Healthy: communicating with pid '424063'
  * fleet-server    (HEALTHY)
                    Healthy: communicating with pid '424069'
  * nginx/metrics   (HEALTHY)
                    Healthy: communicating with pid '424076'
  * filestream      (HEALTHY)
                    Healthy: communicating with pid '424082'
  * beat/metrics    (HEALTHY)
                    Healthy: communicating with pid '424088'
  * http/metrics    (HEALTHY)
                    Healthy: communicating with pid '424094'

Here are the results from the OLD fleet server:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      31924/cupsd         
tcp        0      0 127.0.0.1:8221          0.0.0.0:*               LISTEN      21167/fleet-server  
tcp        0      0 127.0.0.1:6789          0.0.0.0:*               LISTEN      414/elastic-agent   
tcp        0      0 127.0.0.1:6791          0.0.0.0:*               LISTEN      414/elastic-agent   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      482/sshd            
tcp6       0      0 ::1:631                 :::*                    LISTEN      31924/cupsd         
tcp6       0      0 :::8220                 :::*                    LISTEN      21167/fleet-server  
tcp6       0      0 :::22                   :::*                    LISTEN      482/sshd            
udp        0      0 0.0.0.0:58943           0.0.0.0:*                           422/avahi-daemon: r 
udp        0      0 0.0.0.0:68              0.0.0.0:*                           489/dhclient        
udp        0      0 0.0.0.0:631             0.0.0.0:*                           31925/cups-browsed  
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           422/avahi-daemon: r 
udp6       0      0 :::51714                :::*                                422/avahi-daemon: r 
udp6       0      0 :::5353                 :::*                                422/avahi-daemon: r 
root@oldsrv:~# netstat -pln | grep 8220
tcp6       0      0 :::8220                 :::*                    LISTEN      21167/fleet-server

Curl from Agent to old server

curl https://XX.XX.XX.XX:8220/api/status
curl: (28) Failed to connect to XX.XX.XX.XX port 8220: Connection timed out

elastic-agent status
State: HEALTHY
Message: Running
Fleet State: HEALTHY
Fleet Message: Connected
Components:
  * fleet-server  (HEALTHY)
                  Healthy: communicating with pid '21167'
  * filestream    (HEALTHY)
                  Healthy: communicating with pid '21172'
  * beat/metrics  (HEALTHY)
                  Healthy: communicating with pid '21177'
  * http/metrics  (HEALTHY)
                  Healthy: communicating with pid '21179'

Actually I don't know why the curl on old server is timing out.

As this issue is more server / network related (non general issue), I will be glad to thank all who helped especially @MichelLaterman and close this ticket.

Regards,

system · July 10, 2023, 12:33pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.