Load balanced access to fleet servers

fuphil · March 21, 2022, 1:52pm

Hi,
I have setup 2 fleet servers successfully and want all additional elastic agents to connect to them via loadbalanced IP (haproxy with "round robin" option).
Whenever I configure the loadbalanced URL instead of the 2 fleet servers in the fleet settings in kibana, the checkin API call from the elastic agent fails with the following error.
If I use 2 entries with the 2 fleet server urls, everything is fine.

{"log.level":"error","@timestamp":"2022-03-21T12:15:27.195Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post \"https://<my loadbalanced IP>:8220/api/fleet/agents/87908f75-c72f-4a73-9287-bea484589399/checkin?\": EOF","ecs.version":"1.6.0"}

I found this discussion, but it is not clear which scenario is supported at the moment (I am using version 7.16.2). I am trying to setup scenario 1.

github.com/elastic/fleet-server

[Discuss] Using multiple fleet-servers

opened 10:08AM - 19 Nov 21 UTC

ruflin

Team:Elastic-Agent-Control-Plane

Over the past few weeks we have seen several users trying to run multiple fleet-…servers. This issue is to discuss the different scenarios and share an initial thoughts on how we could approach this. The goal is to come a conclusion and feed the information in the documentation and have it guideline for future features. # Core concepts * No load balancing: It is not the Elastic Agent jobs to load balance between multiple fleet-servers. This must be done at the infrastructure level through proxy or DNS. * Fail over: An Elastic Agent supports multiple fleet-server urls as failover. By default the first url is picked. * All info is shared across fleet-servers: Each fleet-server has always the same information no matter where it is deployed. # Scenarios Below is a list of scenarios which should describe the expected behaviours to follow the above core concepts. Not all scenarios are supported today. ## Scenario 1: Multiple fleet-servers, all Elastic Agent connect to all fleet-servers In scenario 1 the users had multiple fleet-servers for redundancy or scale purpose. The user is expected to setup a proxy in front of the fleet-servers or use DNS to access the multiple fleet-servers. In the Fleet UI, a single fleet-server url is used. ## Scenario 2: Elastic Cloud only The user connects all its Elastic Agent to the fleet-server in Elastic Cloud. As ESS already has a proxy in front and allows to spin up redundant fleet-servers, the setup is already as expected today. ## Scenario 3: Elastic Cloud fleet-server and on prem The users is using Elastic Cloud with the fleet-server but also runs its on prem fleet-server to have the fleet-server closer to its Elastic Agents. By default the user wants to have its local fleet-servers to be used. In the Fleet UI, the user puts and additional fleet-server url before the Elastic Cloud fleet-server url. The user defined URL is the one used by default. In case the local fleet-server url is not reachable, Elastic Agents fall back to the Elastic Cloud fleet-server url. The local fleet-server is expected to have the version in sync with the hosted fleet-server. ## Scenario 4: Multi policy with multi data center In this scenario the user has multiple data centers with local fleet-servers and specific policies to each data center. In the Fleet UI, a global fleet-server is specified and in addition a fleet-server per policy can be specified. The fleet-server specified in the policy is the first in the list so it will be picked as the default for all Elastic Agent which are part of the policy.

Can anybody tell me if it is possible to use just one URL to connect to all fleet servers?
It seems like it is not possible at the moment or any elastic agent needs to connect always to the same fleet server?

Thanks in advance!

greetings

mtojek · March 21, 2022, 2:58pm

Hm... the error doesn't look like something is wrong with Fleet Server, but rather with your load balancing solution. Could you please try to configure any Nginx of HAProxy instead (temporarily) and make sure that load-balanced Fleet Servers communicate well with the agent?

fuphil · March 22, 2022, 7:10am

When I call the api status via the loadbalanced address on haproxy, I get a healthy state.
I would assume this is enough to verify the connection, isn't it?

curl https://loadbalancedip:8220/api/status

{"name":"fleet-server","status":"HEALTHY"}

system · April 19, 2022, 9:11am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.