Custom Realm and JarHell

Elastic Stack Elasticsearch
#1

Hello there,
I've attempted to author a custom realm as a fat JAR. When installing it I get the following JarHell error:

[root@iel-dev-tfsr-vm1 x-pack]# ./extension install file:///home/tango/john/ims-realm-1.0.0-SNAPSHOT.zip
-> Downloading file:///home/tango/john/ims-realm-1.0.0-SNAPSHOT.zip
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.AvalonLogger
jar1: /usr/share/elasticsearch/plugins/x-pack/commons-logging-1.1.3.jar
jar2: /usr/share/elasticsearch/plugins/x-pack/extensions/.installing-1737085689021563353/ims-realm-1.0.0-SNAPSHOT.jar
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:187)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.jarHellCheck(InstallXPackExtensionCommand.java:197)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.verify(InstallXPackExtensionCommand.java:169)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.install(InstallXPackExtensionCommand.java:207)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.execute(InstallXPackExtensionCommand.java:116)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.execute(InstallXPackExtensionCommand.java:102)
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.xpack.extensions.XPackExtensionCli.main(XPackExtensionCli.java:36)

Do you know how to trouble-shoot or track down the offending dependency? I have quite a few dependencies:

org.elasticsearch elasticsearch 5.1.2 provided org.elasticsearch.plugin x-pack-api 5.1.2 provided net.java.dev.jna jna 4.2.2 provided org.springframework spring-core 4.3.2.RELEASE compile org.springframework spring-web 4.3.2.RELEASE compile org.springframework.security spring-security-core 4.2.1.RELEASE provided org.apache.logging.log4j log4j-api 2.7 compile org.apache.logging.log4j log4j-core 2.7 compile org.meanbean meanbean 2.0.3 test nl.jqno.equalsverifier equalsverifier 1.7.7 test org.projectlombok lombok 1.14.8 compile com.google.guava guava 18.0 compile com.tango.common.utils tango-common-utils 1.1.0 compile com.tango.common.mvc tango-common-mvc 1.1.3-SNAPSHOT compile com.tango.ims ims-client 2.2.4-SNAPSHOT compile junit junit 4.12 test org.mockito mockito-core 1.10.19 test //John.
#2 
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.AvalonLogger
jar1: /usr/share/elasticsearch/plugins/x-pack/commons-logging-1.1.3.jar
jar2: /usr/share/elasticsearch/plugins/x-pack/extensions/.installing-1737085689021563353/ims-realm-1.0.0-SNAPSHOT.jar

Somewhere in your chain of dependencies you are including commons-logging and then repackaging it into your uberjar.

The best way to fixing that is going depend on what your build process looks like.

If you're using maven, then

 mvn dependency:tree

will show you all your dependencies and how they're being included. Then you could add an explicit exclusion on commons-logging.

Alternatively it might be easier to do it at the stage where you build the uberjar. Just drop any dependencies that are part of x-pack.

#3

Thank you Tim - I've followed your guidance and have successfully installed my custom realm.
My custom realm attempts to authenticate against an external system by sending a HTTP request to that system. On initialization it attempts to create the HTTP client. However, I'm receiving a permissions problem:

Caused by: java.security.AccessControlException: access denied ("java.net.NetPermission" "getProxySelector")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_92]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_92]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_92]
at java.net.ProxySelector.getDefault(ProxySelector.java:94) ~[?:1.8.0_92]
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1147) ~[?:?]
at org.apache.http.impl.client.HttpClients.createSystem(HttpClients.java:66) ~[?:?]
at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.(HttpComponentsClientHttpRequestFactory.java:88) ~[?:?]
at org.springframework.http.client.HttpComponentsAsyncClientHttpRequestFactory.(HttpComponentsAsyncClientHttpRequestFactory.java:67) ~[?:?]
at com.tango.common.mvc.client.AsyncClient.(AsyncClient.java:61) ~[?:?]
at com.tango.common.mvc.client.AsyncClient.(AsyncClient.java:45) ~[?:?]
at com.tango.common.mvc.client.AsyncClient$Builder.build(AsyncClient.java:267) ~[?:?]
at com.tango.xpack.realm.IMSRealm.(IMSRealm.java:67) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:19) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:11) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.doStart(Realms.java:82) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.1.2.jar:5.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_92]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_92]
at org.elasticsearch.node.Node.start(Node.java:542) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:234) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:308) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]

I attempted to add a plugin-security.policy file on my classpath with the following grant -
permission java.net.NetPermission "*";

However, I get the same result? Any ideas how to resolve this one?

#4

I see somebody else had the same issue:

I followed the steps suggested there. Unfortunately, though it doesn't seem to be picking up the permission I specified in x-pack-extension-security.policy. I add code like:

if(sm != null) {
sm.checkPermission(new NetworkPermission("getProxySelector"));
}
AccessController.doPrivileged((PrivilegedAction) () -> {
final AsyncClient asyncClient = new AsyncClient.Builder()
.withConnectionTimeout(connectionTimeout).withSocketTimeout(socketTimeout)
.withMaxConnections(maxConnections).withMaxConnectionsPerHost(maxConnectionsPerHost).build();

this.imsClient = new IMSClient("http://localhost", Optional.of(asyncClient), imsCacheSize, imsCacheExpiryInMinutes);
return null;

});

However, the checkPermission is saying I don't have permission even after the I okayed the permissions when installing the custom realm plugin and the .policy file is present

[root@iel-dev-tfsr-vm1 tango-ims-realm]# pwd
/usr/share/elasticsearch/plugins/x-pack/extensions/tango-ims-realm
[root@iel-dev-tfsr-vm1 tango-ims-realm]# ls
ims-realm-1.0.0-SNAPSHOT.jar x-pack-extension-descriptor.properties x-pack-extension-security.policy

#5

Can you share the full contents of the policy file? Is it the same exact exception?

#6

Thanks for your response.
The contents of my x-pack-extension-descriptor.properties:
description=Tango IMS Realm Extension
version=1.0.0-SNAPSHOT
name=tango-ims-realm
classname=com.tango.xpack.IMSRealmExtension
java.version=1.8
xpack.version=5.1.2

The contents of my x-pack-extension-security.policy:
grant {
permission java.net.NetPermission "getProxySelector";
};

I'm building my fat JAR with maven assembly plugin.
Then when attempting to install I do the following:
[root@iel-dev-tfsr-vm1 x-pack]# ./extension remove tango-ims-realm
-> Removing tango-ims-realm...
[root@iel-dev-tfsr-vm1 x-pack]# ./extension install file:///home/tango/john/tango-ims-realm-1.0.0-SNAPSHOT.zip
-> Downloading file:///home/tango/john/tango-ims-realm-1.0.0-SNAPSHOT.zip
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: x-pack extension requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

Continue with installation? [y/N]y
-> Installed tango-ims-realm
[root@iel-dev-tfsr-vm1 x-pack]# pwd
/usr/share/elasticsearch/bin/x-pack
[root@iel-dev-tfsr-vm1 x-pack]# cd ../../plugins/x-pack/extensions/;chmod 755 *
[root@iel-dev-tfsr-vm1 extensions]# !1667
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch && cd /var/log/elasticsearch/ && tail -f tango-cluster.log
[2017-04-14T08:53:05,484][INFO ][o.e.n.Node ] [] initializing ...
[2017-04-14T08:53:05,554][INFO ][o.e.e.NodeEnvironment ] [vCUgnmd] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [2.8gb], net total_space [18.4gb], spins? [unknown], types [rootfs]
[2017-04-14T08:53:05,554][INFO ][o.e.e.NodeEnvironment ] [vCUgnmd] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-14T08:53:05,599][INFO ][o.e.n.Node ] node name [vCUgnmd] derived from node ID [vCUgnmdpTsevehNdEv9E5A]; set [node.name] to override
[2017-04-14T08:53:05,601][INFO ][o.e.n.Node ] version[5.1.2], pid[36275], build[c8c4c16/2017-01-11T20:18:39.146Z], OS[Linux/3.10.0-229.20.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_92/25.92-b14]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [aggs-matrix-stats]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [ingest-common]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-expression]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-groovy]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-mustache]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-painless]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [percolator]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [reindex]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [transport-netty3]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [transport-netty4]
[2017-04-14T08:53:07,078][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded plugin [x-pack]
[2017-04-14T08:53:09,624][INFO ][o.e.n.Node ] initialized
[2017-04-14T08:53:09,625][INFO ][o.e.n.Node ] [vCUgnmd] starting ...
[2017-04-14T08:53:09,628][ERROR][o.e.b.Bootstrap ] [vCUgnmd] Exception
...
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("jdk.net.NetworkPermission" "getProxySelector")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.2.jar:5.1.2]
Caused by: java.security.AccessControlException: access denied ("jdk.net.NetworkPermission" "getProxySelector")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_92]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_92]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_92]
at com.tango.xpack.realm.IMSRealm.(IMSRealm.java:69) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:19) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:11) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.doStart(Realms.java:82) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.1.2.jar:5.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_92]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_92]
at org.elasticsearch.node.Node.start(Node.java:542) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:234) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:308) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]

#7

It looks like it might boil down to just an incorrect permission name:

Try using jdk.net.NetworkPermission instead of java.net.NetPermission in your policy file.

#8

Jeez - I should have spotted that. Staring at it too long.
This mismatch was caused by my coding:

if(sm != null) {
sm.checkPermission(new NetworkPermission("getProxySelector"));
}

When I changed it to the correct

if(sm != null) {
sm.checkPermission(new NetPermission("getProxySelector"));
}

then the exception lines up with what I have in my grant file.

When I add the line
permission java.net.NetPermission "getProxySelector";

to the file:
/usr/share/elasticsearch/plugins/x-pack/plugin-security.policy

then is seems to work, but I guess this is giving the permission right across x-pack. Should my policy file also be called plugin-security.policy?

#9

This is only granted to your extension.

No, x-pack-extension-security.policy is the correct name.

#10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.