# Elasticsearch JMX debugging

**URL:** <https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880>\
**Category:** Elasticsearch\
**Created:** [December 21, 2018, 5:02pm UTC](https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880 "2018-12-21T17:02:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![cactus](https://avatars.discourse-cdn.com/v4/letter/c/f475e1/32.png) [@cactus](https://discuss.elastic.co/u/cactus)\
**Post date:** [December 21, 2018, 5:02pm UTC](https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880/1 "2018-12-21T17:02:44Z")

</div>

Hello,

I'm trying to debug Elasticsearch with JMX using ldap authentication and jmxremote.access file.

jmxremote.access file:

```auto
testuser readwrite

```

jvm.options file

```auto
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=1111
-Dcom.sun.management.jmxremote.rmi.port=1112
-Dcom.sun.management.jmxremote.ssl=true
-Dcom.sun.management.jmxremote.ssl.need.client.auth=true
-Dcom.sun.management.jmxremote.authenticate=true
-Djava.rmi.server.hostname=1.2.3.4
-Dcom.sun.management.jmxremote.access.file=/jmxmonitoring/jmxremote.access
-Djava.security.auth.login.config=/jmxmonitoring/ldap.config
-Dcom.sun.management.jmxremote.login.config=JMX_elasticsearch
-Djavax.net.ssl.keyStore=/jmxmonitoring/serverkeystore
-Djavax.net.ssl.keyStorePassword=changeit
-Djavax.net.ssl.trustStore=/jmxmonitoring/servertruststore
-Djavax.net.ssl.trustStorePassword=changeit
-Djavax.net.debug=all

```

Ldap config file:

```auto
JMX_elasticsearch {
    com.sun.security.auth.module.LdapLoginModule REQUIRED
        userProvider="ldaps://ldap.com/ou=People,dc=example,dc=com"
        userFilter="(&(uid={USERNAME})(memberof=cn=jmxRemoteAccess,ou=Groups,dc=example,dc=com))"
        authIdentity="uid={USERNAME},ou=People,dc=example,dc=com"
        authzIdentity="{USERNAME}"
        debug=true;
};

```

While connecting to node got the error:

```auto
java.security.AccessControlException: access denied ("javax.management.MBeanPermission" "sun.management.ThreadImpl#-[java.lang:type=Threading]" "getMBeanInfo")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at java.security.AccessController.checkPermission(AccessController.java:884)
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1830)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBeanInfo(DefaultMBeanServerInterceptor.java:1393)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.getMBeanInfo(JmxMBeanServer.java:920)
	at com.sun.jmx.remote.security.MBeanServerAccessController.getMBeanInfo(MBeanServerAccessController.java:399)
	at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1462)
	at javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
	at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1408)
	at javax.management.remote.rmi.RMIConnectionImpl.getMBeanInfo(RMIConnectionImpl.java:905)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
	at sun.rmi.transport.Transport$1.run(Transport.java:200)
	at sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:835)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
	at java.security.AccessController.doPrivileged(Native Method)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
	at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:283)
	at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:260)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161)
	at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
	at javax.management.remote.rmi.RMIConnectionImpl_Stub.getMBeanInfo(Unknown Source)
	at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1079)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.tools.jconsole.ProxyClient$SnapshotInvocationHandler.invoke(ProxyClient.java:992)
	at com.sun.proxy.$Proxy1.getMBeanInfo(Unknown Source)
	at sun.tools.jconsole.ProxyClient.tryConnect(ProxyClient.java:385)
	at sun.tools.jconsole.ProxyClient.connect(ProxyClient.java:313)
	at sun.tools.jconsole.VMPanel$2.run(VMPanel.java:294)

```

Tried to do the same with Logstash debugging using JMX - everything works well. I've used same configs.  
Will be very appreciate if someone could help to understand what I'm doing wrong...

---

<div class="post-metadata">

**Author:** ![jakelandis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jakelandis/32/36163_2.png) [@jakelandis](https://discuss.elastic.co/u/jakelandis)\
**Post date:** [December 21, 2018, 10:46pm UTC](https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880/2 "2018-12-21T22:46:07Z")

</div>

Elasticsearch runs with [Java's Security Manager](https://docs.oracle.com/javase/tutorial/essential/environment/security.html), and JMX is not on the grant list (for good reasons). Logstash does not run with the Security Manager.

Elasticsearch does not expose any custom metrics via JMX MBeans, so even if the Security Manager allowed connections you would only have access to the JVM's MBeans. Much of that information (along with a ton of other good stuff) is exposed via the [\_nodes/stats](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html) endpoint (see the jvm section).

---

<div class="post-metadata">

**Author:** ![cactus](https://avatars.discourse-cdn.com/v4/letter/c/f475e1/32.png) [@cactus](https://discuss.elastic.co/u/cactus)\
**Post date:** [December 22, 2018, 11:00am UTC](https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880/3 "2018-12-22T11:00:08Z")

</div>

Thanks for the answer!  
I want to use jmx to run GC or see what threads are doing in particular moment via jconsole/visualvm.  
I don't want to change any attributes or something like that. My point is to do the same which was described in this article [https://www.elastic.co/guide/en/logstash/5.6/tuning-logstash.html#profiling-the-heap](https://www.elastic.co/guide/en/logstash/5.6/tuning-logstash.html#profiling-the-heap) but with ES.

Interesting thing - I was able to connect to ES node using ssl but without authentication(using next option):

```auto
-Dcom.sun.management.jmxremote.authenticate=false

```

I don't understand why it's not possible to connect to node using any sort of authentication (file-based|ldap)...

**UPD**  
Looks like I should play with this property somehow, no?

```auto
-Djava.security.manager 

```

**UPD2**  
So.....found solution to how make it work:

in jvm.options file:

```auto
-Djava.security.manager 
-Djava.security.policy=jmx.policy

```

in jmx.policy file:

```auto
grant {
    permission java.security.AllPermission "", "";
};

```

and result:

````auto
[LdapLoginModule] user provider: ldaps://ldap.com/ou=People,dc=example,dc=com
[LdapLoginModule] attempting to authenticate user: testuser
[LdapLoginModule] searching for entry belonging to user: testuser
[LdapLoginModule] found entry: uid=testuser,ou=People,dc=example,dc=com
[LdapLoginModule] authentication succeeded
[LdapLoginModule] added LdapPrincipal "uid=testuser,ou=People,dc=example,dc=com" to Subject
[LdapLoginModule] added UserPrincipal "testuser" to Subject```

````

Granting `AllPermission` it's not good idea...maybe should I change it to `permission java.security.MbeanPersmission "","";`? I'm not a specialist in java, if anyone has ideas - please advice 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 19, 2019, 11:00am UTC](https://discuss.elastic.co/t/elasticsearch-jmx-debugging/161880/4 "2019-01-19T11:00:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
