Hi everyone!
Currently we are trying to instrument the Java agent of AppDynamics in a Elasticsearch running on Kubernetes.
We had a few access denied errors when the Appdynamics agent tried to monitor Elasticsearch, but we resolved most with the following policy:
grant codeBase "file:/opt/appdynamics/-" {
permission java.security.AllPermission;
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
permission java.util.PropertyPermission "*", "read,write";
permission java.lang.RuntimePermission "*";
permission java.lang.management.ManagementPermission "monitor";
permission java.lang.reflect.ReflectPermission "suppressAccessChecks";
};
grant {
permission "java.security.SecurityPermission" "*";
permission "java.lang.RuntimePermission" "*";
permission java.io.FilePermission "<<ALL FILES>>","read,write,delete";
permission java.net.SocketPermission "*","accept,connect,resolve,listen";
permission java.util.PropertyPermission "*", "read,write";
permission "java.lang.management.ManagementPermission" "monitor";
permission "java.lang.reflect.ReflectPermission" "*";
permission "javax.management.MBeanServerPermission" "*";
permission "javax.management.MBeanPermission" "*","*";
permission "javax.management.MBeanTrustPermission" "*";
permission java.net.NetPermission "*";
};
However, at times we have the following access denied error that we are unable to resolve:
access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:75)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)
access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:86)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)
access: access denied ("java.lang.RuntimePermission" "getClassLoader")
java.lang.Exception: Stack trace
at java.base/java.lang.Thread.dumpStack(Thread.java:1379)
at java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:462)
at java.base/java.security.AccessController.checkPermission(AccessController.java:1036)
at java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:408)
at java.base/java.lang.ClassLoader.checkClassLoaderPermission(ClassLoader.java:2058)
at java.base/java.lang.Class.getClassLoader(Class.java:836)
at com.appdynamics.appagent/com.singularity.ee.agent.appagent.services.bciengine.transformation.AnonymousClassDefTransformer.classDefTrap(AnonymousClassDefTransformer.java:61)
at com.singularity.ee.agent.appagent.entrypoint.bciengine.AnonymousClassDefTransformerBoot.classDefTrap(AnonymousClassDefTransformerBoot.java:31)
at java.base/jdk.internal.misc.Unsafe.defineAnonymousClass(Unsafe.java:1225)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.spinInnerClass(InnerClassLambdaMetafactory.java:321)
at java.base/java.lang.invoke.InnerClassLambdaMetafactory.buildCallSite(InnerClassLambdaMetafactory.java:189)
at java.base/java.lang.invoke.LambdaMetafactory.metafactory(LambdaMetafactory.java:329)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:127)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
at org.elasticsearch.painless.ScriptClassInfo.methodArgument(ScriptClassInfo.java:180)
at org.elasticsearch.painless.ScriptClassInfo.<init>(ScriptClassInfo.java:99)
at org.elasticsearch.painless.Compiler.compile(Compiler.java:210)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:420)
at org.elasticsearch.painless.PainlessScriptEngine$5.run(PainlessScriptEngine.java:416)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:391)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:416)
at org.elasticsearch.painless.PainlessScriptEngine.compile(PainlessScriptEngine.java:167)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:363)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:148)
at org.elasticsearch.ingest.common.ScriptProcessor$Factory.create(ScriptProcessor.java:90)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:402)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessor(ConfigurationUtils.java:372)
at org.elasticsearch.ingest.ConfigurationUtils.readProcessorConfigs(ConfigurationUtils.java:316)
at org.elasticsearch.ingest.Pipeline.create(Pipeline.java:73)
at org.elasticsearch.ingest.IngestService.innerUpdatePipelines(IngestService.java:515)
at org.elasticsearch.ingest.IngestService.applyClusterState(IngestService.java:259)
at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:484)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:481)
at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:468)
at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:419)
at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:163)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:830)
access: access allowed ("java.security.SecurityPermission" "getPolicy")
access: domain that failed ProtectionDomain null
null
<no principals>
java.security.Permissions@5da5ecc6 (
)
When we access the AppDynamics dashboard, we see that Elasticsearch appears online, but the only metrics captured are CPU and memory usage.
Has anyone experienced this problem or instrumented AppDynamics another way, or can you help solve and try to understand this access denied error?
PS:
- The x-pack-security is currently enabled;
- The AppDynamics Java agent is stored in a volume attached for each Elasticsearch node with read and write access;
- We tried to give access to all this access denied error;
- The java policy we created were applied successfully;
- There is no AppDynamics logs in it's workspace about this access denied error;