Apm java agent server_urls list not balancing

[moixcruz (Moises Cruz)]

Hi! I'm running APM stack all components 6.6.0, and already thinking on upgrade to recently released 7.0.0!

I have mainly java agents (1.6.0) connected to several apm-servers, 3 in test environment.

My java agent is instrumented with properties file:
service_name=my-testing-service
application_packages=hello-world
server_urls=http://apm-server-1.test:8200,http://apm-server-2.test:8200,http://apm-server-3.test:8200

and in some cases I override the servers url via command line -Delastic.apm.server_urls=...

While testing it, I see that agent connects only to first available apm server in the list, apm-server-1.test, and doesn't perform any kind of traffic balance

Looking into documentation https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-server-urls I cannot find any reference to how it is supposed to work, but I was expecting to distribute the traffic across all available servers.

Can anyone explain exactly how it is supposed to work? should I put a reverse proxy on top of all apm-servers to do it?

Many thanks!

[felixbarny (Felix Barnsteiner)]

Hi and welcome to the forum

In order to leverage persistent TCP connections, the Java agent picks a random server url and sticks with it unless this server is under high load or returns a server error. See also https://github.com/felixbarny/apm-agent-java/blob/c9149dbf340c22cd27617f7c55cee7d8b41999db/apm-agent-core/src/main/java/co/elastic/apm/agent/report/IntakeV2ReportingEventHandler.java#L350

Note that the connection test on startup only tests the connection to the first APM Server in the list.

Best,
Felix

[moixcruz (Moises Cruz)]

yeah, has sense, many thanks!