APM-agent-attach-cli-1.29.0 access denied ERROR bytebuddy

Hi,
I am using the apm-agent-attach-cli-1.29.0.jar in order to monitor a Java application (installed openjdk version "1.8.0_292").

Log shows that the agent is not able to start:
[elastic-apm-agent] ERROR Failed to start agent
It looks like the problem is linked with bytebuddy:

Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "net.bytebuddy.createJavaDispatcher")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:886)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at net.bytebuddy.utility.dispatcher.JavaDispatcher.run(JavaDispatcher.java:178)
at java.security.AccessController.doPrivileged(Native Method)
at net.bytebuddy.description.type.TypeDescription$ForLoadedType.doPrivileged(TypeDescription.java)
at net.bytebuddy.description.type.TypeDescription$ForLoadedType.<clinit>(TypeDescription.java:8543)
... 26 more

In the java policy file I added the complete permission for the runtime, but this doesn't fix the permission problem.
permission java.lang.RuntimePermission ".";

Someone has faced this problem?

APM Client version:
apm-agent-attach-cli-1.29.0.jar

APM Agent language and version:
Java

Log:

INFO   | jvm 2    | 2022/03/07 10:27:02.837 | [elastic-apm-agent] ERROR Failed to start agent
INFO   | jvm 2    | 2022/03/07 10:27:02.837 | java.lang.reflect.InvocationTargetException
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:149)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:93)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.premain.AgentMain.agentmain(AgentMain.java:60)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:411)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 | Caused by: java.util.ServiceConfigurationError
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:150)
INFO   | jvm 2    | 2022/03/07 10:27:02.837 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:118)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.<init>(DependencyInjectingServiceLoader.java:69)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.load(DependencyInjectingServiceLoader.java:89)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.loadInstrumentations(ElasticApmAgent.java:168)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.initInstrumentation(ElasticApmAgent.java:160)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:146)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 13 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.lang.reflect.InvocationTargetException
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.util.DependencyInjectingServiceLoader.instantiate(DependencyInjectingServiceLoader.java:141)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 19 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.lang.ExceptionInInitializerError
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.matcher.ElementMatchers.takesArgument(ElementMatchers.java:1248)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at co.elastic.apm.agent.pluginapi.AbstractSpanInstrumentation$InitializeInstrumentation.<init>(AbstractSpanInstrumentation.java:158)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 24 more
INFO   | jvm 2    | 2022/03/07 10:27:02.838 | Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "net.bytebuddy.createJavaDispatcher")
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessController.checkPermission(AccessController.java:886)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.lang.reflect.Method.invoke(Method.java:498)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.utility.dispatcher.JavaDispatcher.run(JavaDispatcher.java:178)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at java.security.AccessController.doPrivileged(Native Method)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.description.type.TypeDescription$ForLoadedType.doPrivileged(TypeDescription.java)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   at net.bytebuddy.description.type.TypeDescription$ForLoadedType.<clinit>(TypeDescription.java:8543)
INFO   | jvm 2    | 2022/03/07 10:27:02.838 |   ... 26 more

Hi @attiliobroglio , welcome to the forum !

Could you elaborate a bit on the need to run the CLI attacher with a Security Manager ?
Does it behaves as expected without the Security Manager ?

From the top of my mind, the attach process will need to be able to do the following:

  • read its own classpath resources to locate the embedded copy of the Elastic APM agent
  • write a copy of the Elastic APM agent to the filesystem (required to attach to the bootstrap classpath).
  • call native code to call the attach API on the target JVM
  • spawn a separate process to list running JVMs, usually the jps command line tool.
  • sometimes spawn a separate JVM when the target JVM is running with a different user.

This is only for the attach process, if a Security Manager is also used on the target JVM, it will also require to adjust configuration to allow the agent to run properly.

If you really need or want to have fine-level control over the attach process, for example because you run it as a privileged user, then you will have to properly configure Security Manager and add permissions until it works. However, you should keep in mind that the attach process will have to run with the same user as the target JVM, so it will use sudo when needed (for example you run the attacher as root and the target JVM is running with another user).

Thus here I would suggest to do the following:

  • try without Security Manager (SM) or the attach process
  • if SM is required for the attach process, you'll have to grant permissions until it works (there might be a lot of them), the general documentation for SM should provide you the details on how to do that.
  • once the attach process works without error, does the target JVM also uses a SM ? If yes, you'll have to also modify the configuration.

Thanks @Sylvain_Juge,
unfortunately in this scenario I cannot disable the SM.

Do you think that try to align my policy to this of elastic (test-framework.policy) could solve my permissions issue?

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.