APM-agent-attach-cli-1.29.0 access denied ERROR bytebuddy

attiliobroglio · March 7, 2022, 11:15am

Hi,
I am using the apm-agent-attach-cli-1.29.0.jar in order to monitor a Java application (installed openjdk version "1.8.0_292").

Log shows that the agent is not able to start:
[elastic-apm-agent] ERROR Failed to start agent
It looks like the problem is linked with bytebuddy:

Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "net.bytebuddy.createJavaDispatcher")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:886)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bytebuddy.utility.dispatcher.JavaDispatcher.run(JavaDispatcher.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at net.bytebuddy.description.type.TypeDescription$ForLoadedType.doPrivileged(TypeDescription.java)
at net.bytebuddy.description.type.TypeDescription$ForLoadedType.<clinit>(TypeDescription.java:8543)
... 26 more

In the java policy file I added the complete permission for the runtime, but this doesn't fix the permission problem.
permission java.lang.RuntimePermission ".";

Someone has faced this problem?

APM Client version:
apm-agent-attach-cli-1.29.0.jar

APM Agent language and version:
Java

Log:

INFO   | jvm 2    | 2022/03/07 10:27:02.837 | [elastic-apm-agent] ERROR Failed to start agent
INFO   | jvm 2    | 2022/03/07 10:27:02.837 | java.lang.reflect.InvocationTargetException
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:149)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:93)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.agentmain(AgentMain.java:60)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 | Caused by: java.util.ServiceConfigurationError
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:150)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:118)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.<init>(DependencyInjectingServiceLoader.java:69)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.load(DependencyInjectingServiceLoader.java:89)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.loadInstrumentations(ElasticApmAgent.java:168)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:160)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:146)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 13 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.lang.reflect.InvocationTargetException
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:141)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 19 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.lang.ExceptionInInitializerError
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.matcher.ElementMatchers.takesArgument(ElementMatchers.java:1248)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.pluginapi.AbstractSpanInstrumentation$InitializeInstrumentation.<init>(AbstractSpanInstrumentation.java:158)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 24 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "net.bytebuddy.createJavaDispatcher")
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessController.checkPermission(AccessController.java:886)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.utility.dispatcher.JavaDispatcher.run(JavaDispatcher.java:178)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessController.doPrivileged(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.description.type.TypeDescription$ForLoadedType.doPrivileged(TypeDescription.java)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.description.type.TypeDescription$ForLoadedType.<clinit>(TypeDescription.java:8543)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 26 more

Sylvain_Juge · March 7, 2022, 1:38pm

Hi @attiliobroglio , welcome to the forum !

Could you elaborate a bit on the need to run the CLI attacher with a Security Manager ?
Does it behaves as expected without the Security Manager ?

From the top of my mind, the attach process will need to be able to do the following:

read its own classpath resources to locate the embedded copy of the Elastic APM agent
write a copy of the Elastic APM agent to the filesystem (required to attach to the bootstrap classpath).
call native code to call the attach API on the target JVM
spawn a separate process to list running JVMs, usually the jps command line tool.
sometimes spawn a separate JVM when the target JVM is running with a different user.

This is only for the attach process, if a Security Manager is also used on the target JVM, it will also require to adjust configuration to allow the agent to run properly.

If you really need or want to have fine-level control over the attach process, for example because you run it as a privileged user, then you will have to properly configure Security Manager and add permissions until it works. However, you should keep in mind that the attach process will have to run with the same user as the target JVM, so it will use sudo when needed (for example you run the attacher as root and the target JVM is running with another user).

Thus here I would suggest to do the following:

try without Security Manager (SM) or the attach process
if SM is required for the attach process, you'll have to grant permissions until it works (there might be a lot of them), the general documentation for SM should provide you the details on how to do that.
once the attach process works without error, does the target JVM also uses a SM ? If yes, you'll have to also modify the configuration.

attiliobroglio · March 7, 2022, 2:07pm

Thanks @Sylvain_Juge,
unfortunately in this scenario I cannot disable the SM.

Do you think that try to align my policy to this of elastic (test-framework.policy) could solve my permissions issue?

github.com

elastic/elasticsearch/blob/master/server/src/main/resources/org/elasticsearch/bootstrap/test-framework.policy

/*
 * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
 * or more contributor license agreements. Licensed under the Elastic License
 * 2.0 and the Server Side Public License, v 1; you may not use this file except
 * in compliance with, at your election, the Elastic License 2.0 or the Server
 * Side Public License, v 1.
 */

//// additional test framework permissions.
//// These are mock objects and test management that we allow test framework libs
//// to provide on our behalf. But tests themselves cannot do this stuff!

grant codeBase "${codebase.mockito-core}" {
  // needed to access ReflectionFactory (see below)
  permission java.lang.RuntimePermission "accessClassInPackage.sun.reflect";
  permission java.lang.RuntimePermission "accessClassInPackage.jdk.internal";
  // needed for reflection in ibm jdk
  permission java.lang.RuntimePermission "accessClassInPackage.sun.misc";
  // needed to support creation of mocks
  permission java.lang.RuntimePermission "reflectionFactoryAccess";

This file has been truncated. show original

system · March 28, 2022, 10:08am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.