Fleet Server not starting with Elastic Agent (Error: fleet-server failed: context canceled)

Hello There!
I am trying to set up my fleet server and elastic agent.
I am asked to do this:

sudo elastic-agent enroll --url=https://{{ip_address}}:{{port_number_1}} \
  --fleet-server-es=https://{{ip_address}}:{{port_number_2}} \
  --fleet-server-service-token=redacted \
  --fleet-server-policy=redacted \
--certificate-authorities=/etc/filebeat/elastic-stack-ca.pem   --fleet-server-es-ca=/etc/elasticsearch/elastic-stack-ca.pem   --fleet-server-cert=/etc/kibana/es.crt.pem   --fleet-server-cert-key=/etc/kibana/es.key.pem

After firing command I am asked am I sure I want to continue:

This will replace your current settings. Do you want to continue? [Y/n]:y

And than this happens:

2022-09-05T10:33:40.599+0200	INFO	cmd/enroll_cmd.go:776	Fleet Server - Stopping
2022-09-05T10:34:42.627+0200	INFO	cmd/enroll_cmd.go:776	Fleet Server - Error - dial tcp 195.187.35.164:8220: connect: connection refused
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html

This is my elastic-agent.yml:

# ================================ General =====================================
# Beats is configured under Fleet, you can define most settings
# from the Kibana UI. You can update this file to configure the settings that
# are not supported by Fleet.
fleet:
  enabled: true

# agent.download:
#   # source of the artifacts, requires elastic like structure and naming of the binari>
#   # e.g /windows-x86.zip
#   sourceURI: "https://artifacts.elastic.co/downloads/beats/"
#   # path to the directory containing downloaded packages
#   target_directory: "${path.data}/downloads"
#   # timeout for downloading package
#   timeout: 120s
#   # file path to a public key used for verifying downloaded artifacts
#   # if not file is present Elastic Agent will try to load public key from elastic.co >
#   pgpfile: "${path.data}/elastic.pgp"
#   # install_path describes the location of installed packages/programs. It is also us>
#   # for reading program specifications.
#   install_path: "${path.data}/install"

# agent.process:
#   # minimal port number for spawned processes
#   min_port: 10000
#   # maximum port number for spawned processes
#   max_port: 30000
#   # timeout for creating new processes. when process is not successfully created by t>
#   # start operation is considered a failure
#   spawn_timeout: 30s

# agent.retry:
#   # enabled determines whether retry is possible. Default is false.
#   enabled: true
#   # retries_count specifies number of retries. Default is 3.
#   # Retry count of 1 means it will be retried one time after one failure.
#   retries_count: 3
#   # delay specifies delay in ms between retries. Default is 30s
#   delay: 30s
#   # max_delay specifies maximum delay in ms between retries. Default is 300s
#   max_delay: 5m
#   # Exponential determines whether delay is treated as exponential.
#   # With 30s delay and 3 retries: 30, 60, 120s
#   # Default is false
#   exponential: false

I tried to compare my error with docs but I can't understand what is wrong.

Hi,

dial tcp 195.187.35.164:8220: connect: connection refused

are you enable to hit this endpoint with telnet?