24 Java files, ElasticAPM Java agent takes more than 6 seconds to attach

[VVK (VVetsa)]

Hello APM Java Gurus

APM Server version:
7.17.6 / 7.17.8
APM Agent language and version:
1.33.0 / 1.34.1

Is there anything special in your setup?
Are you using a load balancer in front of the APM Servers?*
It is running in AWS
Have you changed index pattern, generated custom templates, changed agent configuration etc.
Nothing special
Description of the problem including expected versus actual behaviour. Please include screenshots (if relevant):
My docker image has just 24 java files and fair number of dependencies. It takes approximately 7 seconds to attach.

Provide logs and/or server output (if relevant):

2023-02-06 08:07:48,091 [elastic-apm-metadata-0] INFO  co.elastic.apm.agent.impl.metadata.SystemInfo - Failed to execute command "uname -n" with exit code -1
2023-02-06 08:07:48,171 [elastic-apm-metadata-0] INFO  co.elastic.apm.agent.impl.metadata.SystemInfo - Failed to execute command "hostname" with exit code -1
2023-02-06 08:07:48,311 [Attach Listener] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2023-02-06 08:07:48,406 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.34.1 as XXXXX-YYYY-write-service-svc on Java 11.0.14 Runtime version: 11.0.14+9-post-Debian-1deb11u1 VM version: 11.0.14+9-post-Debian-1deb11u1 (Debian) Linux 5.4.219-126.411.amzn2.x86_64
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'XXXXX-YYYY-write-service-svc' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - environment: 'sandbox' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - transaction_sample_rate: '0.1' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://elk-apm-jaeger-apm-http.elastic-apm:8200' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'AAA.BBB.CCC' (source: Environment Variables)
2023-02-06 08:07:48,407 [Attach Listener] INFO  co.elastic.apm.agent.configuration.StartupInfo - circuit_breaker_enabled: 'true' (source: Environment Variables)
2023-02-06 08:07:57,184 [Attach Listener] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2023-02-06 08:07:57,380 [Service Thread] INFO  co.elastic.apm.agent.micrometer.MicrometerMetricsReporter - Registering Micrometer MeterRegistry: io.micrometer.core.instrument.simple.SimpleMeterRegistry@22d7fd41
2023-02-06 08:07:58,596 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2022-11-24T04:31:09Z",  "build_sha": "311a5edd33673721ad8be4c538635b4d22adeb88",  "publish_ready": true,  "version": "7.17.8"}
2023-02-06 08:07:59,098 [elastic-apm-remote-config-poller] INFO  co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {capture_body=errors, transaction_max_spans=500, transaction_sample_rate=1}

Let me know.. whether I have any issue because of which such slowness to launch agent?
I use the approach discussed in Programmatic API setup to self-attach | APM Java Agent Reference [1.x] | Elastic

Question:
I use springboot. The agent documentation says start the first statement to be executed is ElasticApmAttacher.execute() and then other application code

I tried multiple ways to make the start time less without loosing the effectiveness like in @PostConstruct annotation; ApplicationReadyEvent etc..

Is there any way to reduce the startup time?

What is the problem:
My podSpec's livenessProbe/ReadinessProbes and HPA specs kick in making things messy at the application level.
Meaning: time it takes to launch pod for consumption is quite high and then subsequently issues come because of probes.
one workaround is: increases the threshold values in probes.. but that is not so great solution to convince other applications

[Eyal_Koren]

Take a look at the related guide.

I would start with the third option - setting the enable_instrumentations config. If you do it properly it can make a big difference. In order to get the optimal set of instrumentations to enable, try the following:

1. do increase the timeout threshold temporarily
2. start your application
3. make sure all entry points were invoked, so that all instrumentations are applied
4. stop the service and extract the instrumentation set from the bottom of the agent log
5. use the generated set as the value of the enable_instrumentations config
6. set application_packages with the root package of your 24 Java classes (or anything else that is part of your application and not the dependencies)
7. start your application and see if startup time was reduced enough

If you are using our public API let me know, you may need to add something.

If the above doesn't reduce the startup time enough, try the start_async route.

I hope this helps.
Looking forward to hear back as we are looking into ways of improving our capabilities around startup optimizations.

[VVK (VVetsa)]

Thanks for very quick reply.

I will try to follow your way of looking at the tuning.
As of now I am using only vanilla instrumentation i.e. at the start of application,

• I am just putting ElasticApmAttacher.attach();
• application_packages has been already configured.
• Not yet started using the public api (for span/transaction etc).

I will definitely reply on this thread in coming days