APM Java Agent - AccessControlException

chandra2037 · February 7, 2020, 4:49pm

APM Server version: 7.5.2
APM Agent language and version: Java, 1.12.0
Tomcat Server Version: 8.5.34
JDK Version: 11.0.1

Copying the APM Java agent to the following location while building the docker image

/usr/local/tomcat/apm/apm-agent.jar

Adding the following property to JAVA_OPTS
-javaagent:/usr/local/tomcat/apm/apm-agent.jar

Updated Java policy to AllPermission - ${java.home}/conf/security/java.policy

grant{
permission java.security.AllPermission;
};

Deploying the docker image into Kubernetes. Kubernetes Pod is failing with CrashLoopBackOff, where I am seeing the following errors in the logs.

Seems like JVM able to read the apm-agent jar file, but getting permission issue reading ElasticApm.attached property, I figured this might be something related to APM agent itself?

NOTE: Picked up JDK_JAVA_OPTIONS: --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED
access: access allowed ("java.io.FilePermission" "/usr/local/tomcat/apm/apm-agent.jar" "read")
access: access denied ("java.util.PropertyPermission" "ElasticApm.attached" "read")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1387)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:895)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at java.base/java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1066)
at java.base/java.lang.System.getProperty(System.java:810)
at java.base/java.lang.Boolean.getBoolean(Boolean.java:265)
at co.elastic.apm.agent.bci.AgentMain.init(AgentMain.java:69)
at co.elastic.apm.agent.bci.AgentMain.premain(AgentMain.java:54)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: access allowed ("java.io.FilePermission" "/usr/local/tomcat/apm/apm-agent.jar" "read")
access: domain that failed ProtectionDomain (file:/usr/local/tomcat/apm/apm-agent.jar )
jdk.internal.loader.ClassLoaders$AppClassLoader@512ddf17

java.security.Permissions@7946e1f4 (
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.jasper.runtime.")
("java.lang.RuntimePermission" "accessClassInPackage.com.apple.")
("java.lang.RuntimePermission" "exitVM")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.tomcat")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.jasper.el")
("java.lang.RuntimePermission" "accessClassInPackage.com.sun.beans.")
("java.lang.RuntimePermission" "getAttribute")
("java.lang.RuntimePermission" "accessClassInPackage.com.sun.java.swing.plaf.")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.tomcat.websocket.server")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.catalina.servlet4preview.http")
("java.lang.RuntimePermission" "accessClassInPackage.com.sun.beans")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.catalina.servlet4preview")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.jasper.runtime")
("java.lang.RuntimePermission" "accessClassInPackage.org.apache.tomcat.websocket")
("java.io.FilePermission" "/usr/local/tomcat/apm/apm-agent.jar#plus" "read")
("java.io.FilePermission" "/usr/local/tomcat/apm/apm-agent.jar" "read")
("java.util.PropertyPermission" "java.naming." "read")
("java.util.PropertyPermission" "java.specification.version" "read")
("java.util.PropertyPermission" "java.vm.vendor" "read")
("java.util.PropertyPermission" "path.separator" "read")
("java.util.PropertyPermission" "os.version" "read")
("java.util.PropertyPermission" "java.vendor.url" "read")
("java.util.PropertyPermission" "jaxp.debug" "read")
("java.util.PropertyPermission" "java.vm.name" "read")
("java.util.PropertyPermission" "java.vm.specification.version" "read")
("java.util.PropertyPermission" "os.name" "read")
("java.util.PropertyPermission" "org.apache.catalina.STRICT_SERVLET_COMPLIANCE" "read")
("java.util.PropertyPermission" "java.version" "read")
("java.util.PropertyPermission" "javax.sql." "read")
("java.util.PropertyPermission" "os.arch" "read")
("java.util.PropertyPermission" "java.specification.vendor" "read")
("java.util.PropertyPermission" "java.vm.specification.name" "read")
("java.util.PropertyPermission" "org.apache.tomcat.util.http.ServerCookie.FWD_SLASH_IS_SEPARATOR" "read")
("java.util.PropertyPermission" "java.home" "read")
("java.util.PropertyPermission" "file.separator" "read")
("java.util.PropertyPermission" "line.separator" "read")
("java.util.PropertyPermission" "java.vm.specification.vendor" "read")
("java.util.PropertyPermission" "java.specification.name" "read")
("java.util.PropertyPermission" "java.vendor" "read")
("java.util.PropertyPermission" "java.vm.version" "read")
("java.util.PropertyPermission" "org.apache.tomcat.util.http.ServerCookie.STRICT_NAMING" "read")
("java.util.PropertyPermission" "java.class.version" "read")
)

Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.security.AccessControlException: access denied ("java.util.PropertyPermission" "ElasticApm.attached" "read")
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.base/java.security.AccessController.checkPermission(AccessController.java:895)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at java.base/java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1066)
at java.base/java.lang.System.getProperty(System.java:810)
at java.base/java.lang.Boolean.getBoolean(Boolean.java:265)
FATAL ERROR in native method: processing of -javaagent failed
at co.elastic.apm.agent.bci.AgentMain.init(AgentMain.java:69)
at co.elastic.apm.agent.bci.AgentMain.premain(AgentMain.java:54)
... 6 more

I saw this Github post about security managers, where the user got it working by setting AllPermission. But no luck for me.

Any help will be appreciated.

Thanks,
Chandra

Eyal_Koren · February 9, 2020, 6:36am

Hi and thanks for the question.
Looks like the JVM did not pick up the permissions you attempted to grant. Try this:

grant codeBase "file:/usr/local/tomcat/apm/*" {
    permission java.security.AllPermission;
}

This grants all permissions specifically to the agent.

As long as you keep getting access denied ("java.util.PropertyPermission" "ElasticApm.attached" "read"), it means your policy guideline is not being processed properly so keep trying moving things around. It could be that your policy file is not processed properly, for example - because of how you edited it or because it is not the right policy file (maybe another file's contents are overriding it, although I don't think this is the case in Tomcat).

Note that even once you figure this out, the agent may have other errors related to running with the security manager.

chandra2037 · February 17, 2020, 6:24pm

Hi @Eyal_Koren,

Thank you for your response.

The issue got resolved. You are right that JVM did not pick up the permissions that we configured.

Seems like docker image has the following configuration

-Djava.security.policy==/usr/local/tomcat/conf/catalina.policy

which means the security policy that we are configuration getting overwritten by catalina.policy file. So adding the following statement to the catalina.policy file itself resolved the issue.

grant codeBase "file:/usr/local/tomcat/apm/-" {
     permission java.security.AllPermission;
};

I appreciate your time and help.

Thanks,
Chandra

system · March 9, 2020, 2:24pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.