Load-balancing APM traffic across Elasticsearch nodes

[luisramos (Luis)]

Hello, I've come across https://www.elastic.co/guide/en/kibana/current/production.html#load-balancing-es and believe this is the right approach for our Kibana setup.

We're also setting up APM Server, and was wondering if it is common to also run a coordinator ES node on each APM server host to achieve the same functionality.

What's the recommended approach here for APM - hardcoding the ES node addresses in the APM config or the above?

Thanks in advance!

[axw (Andrew Wilkins)]

Hi @luisramos, welcome to the forum!

I don't think it's at all common for APM Server to have a coordinating ES node running locally.

APM Server is mostly just indexing data, and the few queries to ES it does make are cached. This is unlike Kibana, where queries to Elasticsearch are in the critical path and affect UI responsiveness.

We generally recommend that you position APM Server closer to the Agents to minimise time taken for agents to offload data. That doesn't preclude running a coordinating ES node, but I wouldn't expect it to make a significant difference.

[luisramos (Luis)]

Thanks, @axw - that clarifies my question!

We'll explore moving our APM server closer to our agents as you've recommended - that does make sense.

On a slightly different note, what's your recommendation on which Elasticsearch node types should APM server output to - dedicated ingest or directly to the data nodes? (we already have dedicated ingest nodes which our Beats are using, we're just not sure if there's added benefit of one over the other).

Thanks,

Luis

[axw (Andrew Wilkins)]

On a slightly different note, what's your recommendation on which Elasticsearch node types should APM server output to - dedicated ingest or directly to the data nodes? (we already have dedicated ingest nodes which our Beats are using, we're just not sure if there's added benefit of one over the other).

At the moment, I don't think it would make much difference in general.

Data comes out of APM Server mostly fully formed. We do use ingest node, primarily for GeoIP enrichment and User-Agent parsing. This is mostly relevant for RUM. How heavy the impact on ingest node is depends somewhat on your services, e.g. if you have RUM traffic with many distinct User-Agents, then that would impact User-Agent parsing.

Beats modules (not all, but many) rely more heavily on ingest node pipelines for parsing log messages and so on, so it's more relevant there to scale ingest node independently.