Induced CPU load by APM nodejs agent if elastic server saturates

[Javier_Aracil]

Hi All,

We are instrumenting a nodejs backend with Elastic APM agent. At some point, the elastic server cannot cope with the incoming messages from the agent. I wonder what happens in that case. I understand that such incoming messages will be dropped, and the backend CPU load will not be affected because of this.

Generally speaking, what CPU load can the agent induce in the backend?

Thanks very much,

Javier

[trentm (Trent Mick)]

Hello Javier,

Thanks for the question. I'm assuming by "elastic server" here you mean the APM server (i.e. Configuration options | APM Node.js Agent Reference [4.x] | Elastic) to which the APM agent sends event data.

The first important thing to note is that how the Node.js APM agent copes with an overloaded or non-responding APM server was significantly changed in v3.14.0 of the agent (Node.js Agent version 3.x | APM Node.js Agent Reference [4.x] | Elastic).

@Javier_Aracil

I understand that such incoming messages will be dropped, and the backend CPU load will not be affected because of this.

That is correct (at least with v3.14.0). The APM agent will buffer up to 1024 (Configuration options | APM Node.js Agent Reference [4.x] | Elastic) events and start dropping events if the APM server won't receive them fast enough.

@Javier_Aracil

Generally speaking, what CPU load can the agent induce in the backend?

That is a hard question to quantify in general. Often it depends a lot on the application being instrumented. Some disorganized thoughts:

• The async_hooks mechanism that the agent uses does definitely have some overhead on the application. From recent anecdotal observations, if the application is very heavy on async operations (lots of timers, nextTick, setImmediate, and especially Promise usage), that overhead can be higher.
• If you are experiencing too-high CPU overhead from the agent, my first suggestions would be to consider setting transactionSampleRate (Configuration options | APM Node.js Agent Reference [master] | Elastic) to a value less than 1, and captureSpanStackTraces=false (Configuration options | APM Node.js Agent Reference [master] | Elastic). If the application produces lots of spans, the latter can help a lot. Coming versions of the agent will have some improvements on stacktrace collection overhead and will probably change default settings to not capture stacktraces for all spans (likely just for slower ones).
• Otherwise working through Performance Tuning | APM Node.js Agent Reference [master] | Elastic can help.

I hope that helps answer your question.

Cheers,
Trent

[Javier_Aracil]

Thanks Trent, yes, APM server. We will proceed as suggested, best

Javier