Unable to disable Java APM agent in 1.9.0

Elastic Observability APM
1

We are using the APM agent 1.9.0 in production & pre-prod. For dev services, we want to disable the APM, for this we are trying to set via Environment variables. Disabling of APM worked in 1.7.0, but it is not working in 1.8.0 & 1.9.0.

Here are the kubernetes pod details,

Environment:
PROFILE: dev
ELASTIC_APM_ACTIVE: false
ELASTIC_APM_INSTRUMENT: false

elasticapm.properties:

service_name=task-service
application_packages=com.jay
disable_instrumentations=none
transaction_sample_rate=0.1

With the above configuration still the agent is active. Here are the trace logs,

2019-09-02 10:01:34.093 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Found Kubernetes pod UID: a2c41c12-cd68-11e9-bcdf-02daf193ac9c
2019-09-02 10:01:34.094 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - container ID is ee74a43ea8010aa6719fca3a712e0e443bc677e4a37743dfd6ecf7eb30074506
2019-09-02 10:01:34.112 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.9.0 as task-service on Java 1.8.0_212 (Oracle Corporation) Linux 4.9.0-9-amd64
2019-09-02 10:01:34.117 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - active: 'false' (source: Environment Variables)
2019-09-02 10:01:34.117 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - instrument: 'false' (source: Environment Variables)
2019-09-02 10:01:34.117 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - transaction_sample_rate: '0.1' (source: //elasticapm.properties)
2019-09-02 10:01:34.117 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - disable_instrumentations: 'none' (source: //elasticapm.properties)
2019-09-02 10:01:34.117 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.jay' (source: //elasticapm.properties)
2019-09-02 10:01:34.118 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'TRACE' (source: Environment Variables)
2019-09-02 10:01:34.196 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server http://localhost:8200/config/v1/agents
2019-09-02 10:01:34.196 [apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to http://localhost:8200/
2019-09-02 10:01:34.200 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.report.ApmServerClient - Exception while interacting with APM Server, trying next one.
2019-09-02 10:01:34.200 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused (Connection refused)
2019-09-02 10:01:34.200 [apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server http://localhost:8200/ is not available (Connection refused (Connection refused))
2019-09-02 10:01:34.200 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2019-09-02 10:01:34.219 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$StartTransactionInstrumentation
2019-09-02 10:01:34.224 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$StartTransactionWithRemoteParentInstrumentation
...
2019-09-02 10:01:34.233 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying instrumentation co.elastic.apm.agent.opentracing.impl.ElasticApmTracerInstrumentation
2019-09-02 10:01:34.233 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applied 53 advices

2019-09-02 10:06:34.201 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Reloading configuration from APM Server http://localhost:8200/config/v1/agents
2019-09-02 10:06:34.201 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.report.ApmServerClient - Exception while interacting with APM Server, trying next one.
2019-09-02 10:06:34.201 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused (Connection refused)
2019-09-02 10:06:34.201 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s

How to disable the agent? The config says these are the options, but still these are not working.

2

Hi and thanks for the question.

I agree it is not as straightforward as it may be to understand what these options entirely do, so I will try to make some sense of it.

Firstly, the active config option doesn't entirely make the agent 100% noop. The reason is that this config option is a dynamic one, which means we want to make it reversible, so it still needs to do some minimal stuff. Specifically, in 1.8.0 we added our end of the remote agent configuration feature. At the moment, it only supports the transaction_sample_rate config, but it is going to control other dynamic configuration options, very likely that one of the upcoming ones will be the active config :slight_smile: . Therefore, the agent still needs to maintain minimal communication with the server, which is why you see the healthcheck messages and the 5 min scheduling. If you don't want the agent to do even that, try setting the server_urls config option to an empty string.

The instrument config option means that no auto-tracing will take place. Still, you can use the API and the OpenTracing bridge when this option is set to false, which is why you see we apply co.elastic.apm.agent.plugin.api and co.elastic.apm.agent.opentracing advices.

In terms of functionality, the agent has minimal-to-no effects on your application with the configurations you use. You can clean the logs if that is a problem by setting the log level to ERROR, as it currently seem to be set to TRACE.

I hope this makes sense.
Eyal.

3

Now after making server_url as empty string and sample rate to 0, getting following logs,

2019-09-03 09:18:44.135 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - active: 'false' (source: Environment Variables)
2019-09-03 09:18:44.135 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Expected previousException not to be null
2019-09-03 09:18:44.135 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - instrument: 'false' (source: Environment Variables)
2019-09-03 09:18:44.135 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2019-09-03 09:18:44.135 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - transaction_sample_rate: '0.0' (source: Environment Variables)
2019-09-03 09:18:44.135 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - disable_instrumentations: 'none' (source: //elasticapm.properties)
2019-09-03 09:18:44.136 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - server_urls: '' (source: Environment Variables)

2019-09-03 09:23:44.135 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Expected previousException not to be null
2019-09-03 09:23:44.136 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s
2019-09-03 09:28:44.136 [apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Expected previousException not to be null
2019-09-03 09:28:44.136 [apm-remote-config-poller] DEBUG co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Scheduling next remote configuration reload in 300s

Is there anyway to configure it to be a noop.

4

You can also set the central_config option to false.

5

Thanks. After setting central_config to false, getting only DEBUG logs.

6

Thanks for confirming.
I did a small fix so that we do not print this error when server_urls list is empty.

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.