Elasticsearch JMX debugging

[cactus (Roman)]

Hello,

I'm trying to debug Elasticsearch with JMX using ldap authentication and jmxremote.access file.

jmxremote.access file:

testuser readwrite

jvm.options file

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1111
-Dcom.sun.management.jmxremote.rmi.port=1112
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true
-Dcom.sun.management.jmxremote.authenticate=true
-Djava.rmi.server.hostname=1.2.3.4
-Dcom.sun.management.jmxremote.access.file=/jmxmonitoring/jmxremote.access
-Djava.security.auth.login.config=/jmxmonitoring/ldap.config
-Dcom.sun.management.jmxremote.login.config=JMX_elasticsearch
-Djavax.net.ssl.keyStore=/jmxmonitoring/serverkeystore
-Djavax.net.ssl.keyStorePassword=changeit
-Djavax.net.ssl.trustStore=/jmxmonitoring/servertruststore
-Djavax.net.ssl.trustStorePassword=changeit
-Djavax.net.debug=all

Ldap config file:

JMX_elasticsearch {
    com.sun.security.auth.module.LdapLoginModule REQUIRED
        userProvider="ldaps://ldap.com/ou=People,dc=example,dc=com"
        userFilter="(&(uid={USERNAME})(memberof=cn=jmxRemoteAccess,ou=Groups,dc=example,dc=com))"
        authIdentity="uid={USERNAME},ou=People,dc=example,dc=com"
        authzIdentity="{USERNAME}"
        debug=true;
};

While connecting to node got the error:

java.security.AccessControlException: access denied ("javax.management.MBeanPermission" "sun.management.ThreadImpl#-[java.lang:type=Threading]" "getMBeanInfo")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1830)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1393)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)
	at com.sun.jmx.remote.security.MBeanServerAccessController.getMBeanInfo(MBeanServerAccessController.java:399)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1462)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
	at javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:905)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:835)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
	at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl_Stub.getMBeanInfo(Unknown Source)
	at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1079)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.tools.jconsole.ProxyClient$SnapshotInvocationHandler.invoke(ProxyClient.java:992)
	at com.sun.proxy.$Proxy1.getMBeanInfo(Unknown Source)
	at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:385)
	at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
	at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:294)

Tried to do the same with Logstash debugging using JMX - everything works well. I've used same configs.
Will be very appreciate if someone could help to understand what I'm doing wrong...

[jakelandis]

Elasticsearch runs with Java's Security Manager, and JMX is not on the grant list (for good reasons). Logstash does not run with the Security Manager.

Elasticsearch does not expose any custom metrics via JMX MBeans, so even if the Security Manager allowed connections you would only have access to the JVM's MBeans. Much of that information (along with a ton of other good stuff) is exposed via the _nodes/stats endpoint (see the jvm section).

[cactus (Roman)]

Thanks for the answer!
I want to use jmx to run GC or see what threads are doing in particular moment via jconsole/visualvm.
I don't want to change any attributes or something like that. My point is to do the same which was described in this article https://www.elastic.co/guide/en/logstash/5.6/tuning-logstash.html#profiling-the-heap but with ES.

Interesting thing - I was able to connect to ES node using ssl but without authentication(using next option):

-Dcom.sun.management.jmxremote.authenticate=false

I don't understand why it's not possible to connect to node using any sort of authentication (file-based|ldap)...

UPD
Looks like I should play with this property somehow, no?

-Djava.security.manager 

UPD2
So.....found solution to how make it work:

in jvm.options file:

-Djava.security.manager 
-Djava.security.policy=jmx.policy

in jmx.policy file:

grant {
    permission java.security.AllPermission "", "";
};

and result:

[LdapLoginModule] user provider: ldaps://ldap.com/ou=People,dc=example,dc=com
[LdapLoginModule] attempting to authenticate user: testuser
[LdapLoginModule] searching for entry belonging to user: testuser
[LdapLoginModule] found entry: uid=testuser,ou=People,dc=example,dc=com
[LdapLoginModule] authentication succeeded
[LdapLoginModule] added LdapPrincipal "uid=testuser,ou=People,dc=example,dc=com" to Subject
[LdapLoginModule] added UserPrincipal "testuser" to Subject```

Granting AllPermission it's not good idea...maybe should I change it to permission java.security.MbeanPersmission "","";? I'm not a specialist in java, if anyone has ideas - please advice