How to send apm agent data to two different apm-servers

Elastic Observability APM
1

I am using apm.agent 1.5 to a run a simple java app (basically just prints out hello world when you curl at a few endpoints) and send that apm data to an apm-server 6.7.0. The apm-server forwards the data to two separate AWS ES clusters 7.1

I have filled out
-Delastic.apm.server_url=http://url1:8200

and the above works perfectly fine to send to a single cluster

-Delastic.apm.server_urls=http://url1:8200,http://url2:8200
-Delastic.apm.server_urls=http://url1:8200,http://url2:8200.

Neither of the above two works. I have been digging online for examples of how to use the elastic.apm.server_urls input but none of them seem to properly send to both. Only one or the other of the clusters

thanks!

2

Hi @Ashwin_Iyer,

The server_urls configuration option only achieves load-balancing over the provided URLs in case of network or server error, which means data is only sent to a single APM server instance, not both of them.

The agent shuffles the list, thus it provides load-balancing when many agents are deployed with the same list. Changing the list order has no effect.

Also, I'm not sure to understand your use case here, why do you need to send data to multiple server instances ? Is that required to perform server upgrades ?

3

well the goal was the have a steady stream of data from lets say app123 to apm server url 1 to es cluster 1
there would be a secondary stream of apm data from app123 to apm server url 2 to es cluster 2

the apm server2 and es cluster 2 could be utilized to play with configuration settings and specific logging settings. we could also load test the apm-server with specific bulk_max_size settings etc.

apm server 1 and es cluster 1 would be used for our actual data that our team could use to analyze their apps' performance in the meanwhile

so to answer your question; the goal is to create a secondary environment that could be rigorously load tested and reliably allowed to fail. makes sense?

EDIT: so the answer is "no, apm-server-urls does not support sending to both"
https://www.elastic.co/guide/en/apm/agent/dotnet/current/config-reporter.html
Providing multiple URLs is not supported by the agent yet. If multiple URLs are provided only the first one will be used.

thanks and let me know if there are alternatives

4

That's definitely an interesting idea, what you need is basically a reverse HTTP proxy that replicates part of its traffic to another backend server.

I've found a few open-source projects that seem to allow to do just that:

I haven't tested any of them, thus I don't know what maturity level those projects are.
But at least that should provide you a few ideas to implement that.

5

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.