Unable to disable APM via environment variable

jaks · May 10, 2019, 12:35pm

We are using the APM in production & pre-prod. For dev services, we want to disable the APM, for this we are trying to set via Environment variables. Currently using Java agent 1.6.1.

Here are the kubernetes pod details,

Environment:
PROFILE: dev
ELASTIC_APM_ACTIVE: false
ELASTIC_APM_INSTRUMENT: false

elasticapm.properties:

service_name=super-user-service
application_packages=com.jay
disable_instrumentations=none
transaction_sample_rate=0.1

With the above configuration agent is active,

2019-05-10 17:56:08.767 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.6.1 as super-user-service on Java 1.8.0_201 (Oracle Corporation) Linux 4.4.148-k8s
2019-05-10 17:56:09.364 [apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is not available (Connection refused (Connection refused))
2019-05-10 17:56:39.059 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-10 17:56:39.059 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)

How to disable via environment variable? because this way, code or other property files need not be changed

felixbarny · May 10, 2019, 12:49pm

Hi and thanks for the question. Could you send me the debug logs from the startup?

jaks · May 10, 2019, 1:20pm

@felixbarny Here are the debug logs,

2019-05-10 18:41:26.961 [apm-server-healthcheck] DEBUG co.elastic.apm.agent.report.ApmServerHealthChecker - Starting healthcheck to http://localhost:8200/
2019-05-10 18:41:27.269 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - Found Kubernetes pod UID: 1be1f96c-7325-11e9-bdde-023344833348
2019-05-10 18:41:27.270 [main] DEBUG co.elastic.apm.agent.impl.payload.SystemInfo - container ID is 85ff941bd2f32b8cd986cf9608d4236a123180d139564a8fbcd7f17e467e5fe3
2019-05-10 18:41:27.467 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.6.1 as super-user-service on Java 1.8.0_201 (Oracle Corporation) Linux 4.4.148-k8s
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - active: 'false' (source: Environment Variables)
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - instrument: 'false' (source: Environment Variables)
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - transaction_sample_rate: '0.1' (source: //elasticapm.properties)
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - disable_instrumentations: 'none' (source: //elasticapm.properties)
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.jay' (source: //elasticapm.properties)
2019-05-10 18:41:27.468 [main] DEBUG co.elastic.apm.agent.configuration.StartupInfo - log_level: 'DEBUG' (source: //elasticapm.properties)
2019-05-10 18:41:27.866 [apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is not available (Connection refused (Connection refused))
2019-05-10 18:41:28.060 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$StartTransactionInstrumentation
2019-05-10 18:41:28.066 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$StartTransactionWithRemoteParentInstrumentation
2019-05-10 18:41:28.067 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$CurrentTransactionInstrumentation
2019-05-10 18:41:28.067 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$CurrentSpanInstrumentation
2019-05-10 18:41:28.067 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.ElasticApmApiInstrumentation$CaptureExceptionInstrumentation
2019-05-10 18:41:28.067 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.TransactionInstrumentation$SetUserInstrumentation
2019-05-10 18:41:28.067 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.TransactionInstrumentation$EnsureParentIdInstrumentation
2019-05-10 18:41:28.068 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.TransactionInstrumentation$SetResultInstrumentation
2019-05-10 18:41:28.068 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.AbstractSpanInstrumentation$SetNameInstrumentation
2019-05-10 18:41:28.068 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.AbstractSpanInstrumentation$SetTypeInstrumentation
2019-05-10 18:41:28.068 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.AbstractSpanInstrumentation$SetTypesInstrumentation
2019-05-10 18:41:28.068 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applying advice co.elastic.apm.agent.plugin.api.AbstractSpanInstrumentation$DoCreateSpanInstrumentation

...

2019-05-10 18:41:28.078 [main] DEBUG co.elastic.apm.agent.bci.ElasticApmAgent - Applied 47 advices
2019-05-10 18:41:57.761 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving METRICS event (sequence 0)
2019-05-10 18:41:57.761 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Starting new request to http://localhost:8200/intake/v2/events
2019-05-10 18:41:57.762 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)

Eyal_Koren · May 13, 2019, 1:13pm

The active and instrument flags are mostly used to disable instrumentation.
In this case, the agent still tries to establish a connection with the APM server and collect and send metrics. I believe this shouldn't affect your dev env, it only writes to the log.

If you add ELASTIC_APM_METRICS_INTERVAL: 0s (see config option documentation), metrics collection will be disabled.
In addition, you can add ELASTIC_APM_LOG_LEVEL: ERROR to remove agent loggings.

This doesn't mean no agent code will run at all, for that you need to remove the agent altogether. However, it may be enough for your purpose.

I hope this helps.
Eyal.

jaks · May 14, 2019, 3:01pm

Thanks. I was thinking it will not try to connect. Will enable these flags

yycm · May 27, 2019, 1:52am

2019-05-27 09:27:16.989 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:27:16.989 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:27:51.275 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:27:51.275 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:28:29.350 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:28:29.350 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:29:06.018 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:29:06.018 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:29:39.416 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:29:39.417 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:30:15.105 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:30:15.105 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:30:51.391 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:30:51.391 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:31:24.047 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:31:24.047 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2019-05-27 09:31:58.996 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-05-27 09:31:58.996 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

felixbarny · May 27, 2019, 7:45am

We have created an issue for this: https://github.com/elastic/apm-agent-java/issues/639

yycm · May 28, 2019, 9:38am

thanks,felixbarny!!!!

jaks · June 14, 2019, 9:37am

@felixbarny whether this issue is covered as part of agent 1.7.0 release?

felixbarny · June 14, 2019, 10:00am

Yes, this is fixed in 1.7.0. Let me know how it goes.

jaks · June 14, 2019, 11:19am

Thanks. Verified with 1.7.0 and its working

system · July 5, 2019, 7:19am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.