APM Agent High CPU Load

Elastic Observability APM
1

I am trying to instrument a SpringBoot application with the Java APM agent.
The SpringBoot application runs in a container based on the arm64v8/adoptopenjdk image. The APM agent is attached with the -javaagent method in the Docker CMD. SERVER_URLS, SERVICE_NAME, ENVIRONMENT, APPLICATION_PACKAGES, LOG_FILE and LOG_LEVEL are configured through environment variables. Additionally, I disabled all instrumentations by adding all possible options to -Delastic.apm.disable_instrumentations.
While the startup of the SpringBoot usually takes about 60ms, attaching the APM agent causes the startup to take over 50s. This is caused by an extremely high CPU usage of 100%, which starts as soon as the APM agent starts.

Is this a problem with the agent running on ARM?

Other Go services that run on the same node do not experience any of these CPU issues.

Kibana version: 7.7

Elasticsearch version: 7.7

APM Server version: 7.7

APM Agent language and version: Java, 1.16

Browser version: Irrelevant

Original install method (e.g. download page, yum, deb, from source, etc.) and version: https://search.maven.org/remotecontent?filepath=co/elastic/apm/elastic-apm-agent/1.16.0/elastic-apm-agent-1.16.0.jar

Fresh install or upgraded from other version? Fresh install

Is there anything special in your setup? The Java SpringBoot application is running in an arm64 container.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. Attach agent to SpringBoot application running in arm64v8/adoptopenjdk based container

Errors in browser console (if relevant): None

Provide logs and/or server output (if relevant): Logs: https://gist.github.com/niklaskhf/b8fcd36b29d8d2a1e10a4338ff376435

2

That's definitely strange and unexpected. Especially since you have disabled all instrumentations.
According to the logs, the agent starts up in under one second.

Could you get some thread dumps or attach a profiler at the startup?

3

I took some thread dumps during various stages of the startup:

After the startup eventually finishes, the CPU load goes down from 100% to ~20%, which is still higher than without the agent.

Edit: These are with the instrumentation enabled, resulting in a startup time of about 140s.

4

What's the normal baseline?

I couldn't spot any obvious culprit. Maybe park/unpark is especially expensive with ARM and Java 8?

What's the CPU you're running it on? Is there a chance you could try with a more recent Java version?

5

The usual idle load is ~8%. I will try to run this setup on different instance type / Java version configurations and report back if I find anything wrong with the ARM / Java 8 version in particular. It's running on an a1.medium AWS instance. I have a feeling that upgrading the instance type and CPU might already help with this.

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.