Java Automatic Agent Configuration Options are not working

Kevin_Christopher · April 22, 2021, 9:51am

Kibana version:
7.12

Elasticsearch version:
7.12

APM Server version:
7.12

APM Agent language and version:
Java
1.21

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
From Source
Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
No

Curl to apm server url returns 200

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Automatic Java agent cannot pick the server_url given in elasticapm.properties or dynamic

Steps to reproduce:
1.sudo java -jar apm-agent-attach-standalone.jar --include '.sonar.' --config server_url=https://elk:8200
2.sudo java -jar apm-agent-attach-standalone.jar --include '.sonar.' --config config_file=elasticapm.properties
3.sudo java -jar apm-agent-attach-standalone.jar --pid 12898 --config server_url=https://elk:8200
3.Agent is not taking the server_url input

Errors in browser console (if relevant):
No
In console INFO is showing as attached

Provide logs and/or server output (if relevant):

s (+/-10%)
2021-04-22 09:34:22,774 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server. Some details about SSL configurations corresponding the current connection are logged at INFO level.
2021-04-22 09:34:22,774 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: Connection refused (Connection refused)
2021-04-22 09:34:22,774 [elastic-apm-server-reporter] INFO  co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 36 seconds (+/-10%)
2021-04-22 09:34:25,709 [elastic-apm-configuration-reloader] WARN  co.elastic.apm.agent.shaded.stagemonitor.configuration.ConfigurationOption - Error in Attachment configuration: no protocol: 8200 Default value 'http://localhost:8200' for 'server_url' will be applied.
2021-04-22 09:34:31,969 [elastic-apm-configuration-reloader] WARN  co.elastic.apm.agent.shaded.stagemonitor.configuration.ConfigurationOption - Error in Attachment configuration: no protocol: 8200 Default value 'http://localhost:8200' for 'server_url' will be applied.
2021-04-22 09:34:38,425 [elastic-apm-configuration-reloader] WARN  co.elastic.apm.agent.shaded.stagemonitor.configuration.ConfigurationOption - Error in Attachment configuration: no protocol: 8200 Default value 'http://localhost:8200' for 'server_url' will be applied.
2021-04-22 09:34:46,823 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server. Some details about SSL configurations corresponding the current connection are logged at INFO level.
2021-04-22 09:34:46,823 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type SPAN with this error: Connection refused (Connection refused)

For using include option to attach agent to sonar:

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appendedFailed to start agent
java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")        at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.base/java.security.AccessController.checkPermission(AccessController.java:897)        at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
        at java.base/java.lang.Class.forName(Class.java:393)        at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:174)
        at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:122)        at co.elastic.apm.agent.premain.AgentMain.agentmain(AgentMain.java:67)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)        at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:535)
2021-04-22 10:29:15,894 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Connection refused (Connection refused)
2021-04-22 10:29:25,709 [elastic-apm-configuration-reloader] WARN  co.elastic.apm.agent.shaded.stagemonitor.configuration.ConfigurationOption - Error in Attachment configuration: no protocol: 8200 Default value 'http://localhost:8200' for 'server_url' will be applied.
2021-04-22 10:29:25,868 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error trying to connect to APM Server. Some details about SSL configurations corresponding the current connection are logged at INFO level.
2021-04-22 10:29:25,868 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: Connection refused (Connection refused)

Note: I cant use manual attach setup

Eyal_Koren · April 22, 2021, 12:31pm

Hi and thanks for the question!

Not sure what this is about, looks like the agent gets only 8200 for the server_url config. Maybe a matter of escaping. However, before you proceed with your trials, I suggest you upgrade to 1.23.0 and switch to using the new cli. This migration would be recommended anyway, so maybe you'll figure it out while doing so. Just make sure to review the related documentation.

With this version, your non-default configurations are also printed to the log at INFO level, so you should see what the effective server_url is with the default log level.

Note: I cant use manual attach setup

This looks like a security-manager (AKA Java 2 security) error. If you are enabling this mechanism, you would have to grant permissions to the agent jar in your JVM policy.

Kevin_Christopher · April 22, 2021, 2:41pm

Hi @Eyal_Koren

For Upgrading the agent:
We just need to download the latest 1.23.0 jar
And execute the same command right?

Note: I dont want to shutdown and restart the application..

Not sure what this is about, looks like the agent gets only 8200

This is because I have started the agent first time with just 8200 by mistake

you would have to grant permissions to the agent jar in your JVM policy

Ohhh...so If I grant the permission for agent jar.. Will it impact any running jvms like sonar?

Eyal_Koren · April 25, 2021, 5:04am

Follow the documentation. It's a different Java executable (apm-agent-attach-cli.jar) and configuration is done a bit differently.

Then only apply on your next restart.

Depending on your app/JVM, you need to update the right policy file. Typically, those are only loaded at startup, so it would not affect already running JVMs.

Kevin_Christopher · April 25, 2021, 7:53am

@Eyal_Koren

Thanks for the valuable information.

I had to restart the application and attach the agent again.. It worked
I have used 1.22v

system · May 16, 2021, 3:53am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.