Log Messages despite recording=false and instrument=false

sopgreg · June 24, 2020, 1:10pm

APM Agent language and version: Java, 1.17

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Starting JBoss with -javaagent with recording = false and instrument = false still outputs error messages each few seconds. There is no APM server running as we only want to enable recording and instrumentation on demand. Is this the correct behaviour?

Steps to reproduce:

Start APM agent with -javaagent and recording = false and instrument = false
Wait a few seconds for the messages to appear

Provide logs and/or server output (if relevant):

2020-06-24 14:56:35,522 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.17.0 as JBoss on Java 11.0.7 (Azul Systems, Inc.) Windows 10 10.0
2020-06-24 14:56:36,615 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server http://localhost:8200/ is not available (Connection refused: connect)
2020-06-24 14:56:37,349 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to PAUSED state
2020-06-24 14:58:43,460 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:58:43,461 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
2020-06-24 14:58:44,941 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:58:44,942 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (+/-10%)
2020-06-24 14:58:46,909 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:58:46,912 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 4 seconds (+/-10%)
2020-06-24 14:58:51,908 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:58:51,908 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 9 seconds (+/-10%)
2020-06-24 14:59:01,660 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:59:01,661 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 16 seconds (+/-10%)
2020-06-24 14:59:17,647 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
2020-06-24 14:59:17,648 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 25 seconds (+/-10%)
2020-06-24 15:06:41,636 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused: connect

felixbarny · June 24, 2020, 2:49pm

Are you using the API to capture errors? Currently, we do try to send errors even if recording = false but it seems like a bug to me.

However, even after fixing that, the agent will make requests to APM Server every 30s to get remote configuration updates. That ensures that you can set recording to true via remote configuration:

You can disable that by setting central_config=false.

sopgreg · June 24, 2020, 3:49pm

Yes, we do capture some errors using ElasticApm.currentSpan()/currentTransaction()#captureException

Okay, I set central_config=false as we don't need central updates. Btw. what does central_config overwrite on host side? I mean, it shouldn't overwrite things like service_name etc. Can you add some more documentation there?

I restarted completely, this is what we get now:

recording=false
instrument=false
central_config=false

    2020-06-24 17:42:41,675 [main] DEBUG co.elastic.apm.agent.jmx.JmxMetricTracker - Deferring initialization of JMX metric tracking until log manager is initialized
    2020-06-24 17:42:41,680 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to PAUSED state    
    2020-06-24 17:42:49,948 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused: connect
    2020-06-24 17:42:50,073 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (+/-10%)
    2020-06-24 17:43:19,933 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused: connect
    2020-06-24 17:43:19,980 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (+/-10%)
    2020-06-24 17:43:49,952 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused: connect
    2020-06-24 17:43:49,987 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 4 seconds (+/-10%)
    2020-06-24 17:44:19,898 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused: connect
    2020-06-24 17:44:19,899 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 9 seconds (+/-10%)
    2020-06-24 17:44:29,917 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused: connect
    2020-06-24 17:44:29,918 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 16 seconds (+/-10%)
    2020-06-24 17:44:45,892 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused: connect
    2020-06-24 17:44:45,923 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 25 seconds (+/-10%)
    2020-06-24 17:44:53,897 [ComFW-Server-Monitor-Thread-1] INFO  co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
    2020-06-24 17:44:53,951 [ComFW-Server-Monitor-Thread-1] INFO  co.elastic.apm.agent.bci.bytebuddy.CustomElementMatchers - Cannot read implementation version based on ProtectionDomain. This should not affect your agent's functionality. Failed with message: URI scheme is not "file"
    2020-06-24 17:45:12,525 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused: connect
    2020-06-24 17:45:12,525 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
    2020-06-24 17:51:19,368 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type ERROR with this error: Connection refused: connect
    2020-06-24 17:51:19,378 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)

Right now, we only seem to get the "Faile to handle event of type ERROR" messages so far.

felixbarny · June 25, 2020, 7:36am

Could you try out the artifacts from this merged PR that fixes the issue? https://github.com/elastic/apm-agent-java/pull/1252#issuecomment-648894127

sopgreg · July 3, 2020, 6:24am

I see this PR was already merged into master. I'm going to try the new artifact once the PR for this fix is merged as well: No JMX metrics in JBoss EAP 7.3.0

Then I'm also going to play around a bit with the new "enabled" flag in master.

system · July 24, 2020, 2:24am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.