Custom Realm and JarHell

jhn134910 · April 12, 2017, 3:53pm

Hello there,
I've attempted to author a custom realm as a fat JAR. When installing it I get the following JarHell error:

[root@iel-dev-tfsr-vm1 x-pack]# ./extension install file:///home/tango/john/ims-realm-1.0.0-SNAPSHOT.zip
-> Downloading file:///home/tango/john/ims-realm-1.0.0-SNAPSHOT.zip
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.AvalonLogger
jar1: /usr/share/elasticsearch/plugins/x-pack/commons-logging-1.1.3.jar
jar2: /usr/share/elasticsearch/plugins/x-pack/extensions/.installing-1737085689021563353/ims-realm-1.0.0-SNAPSHOT.jar
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:277)
at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:187)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.jarHellCheck(InstallXPackExtensionCommand.java:197)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.verify(InstallXPackExtensionCommand.java:169)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.install(InstallXPackExtensionCommand.java:207)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.execute(InstallXPackExtensionCommand.java:116)
at org.elasticsearch.xpack.extensions.InstallXPackExtensionCommand.execute(InstallXPackExtensionCommand.java:102)
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:69)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122)
at org.elasticsearch.cli.Command.main(Command.java:88)
at org.elasticsearch.xpack.extensions.XPackExtensionCli.main(XPackExtensionCli.java:36)

Do you know how to trouble-shoot or track down the offending dependency? I have quite a few dependencies:

org.elasticsearch elasticsearch 5.1.2 provided org.elasticsearch.plugin x-pack-api 5.1.2 provided net.java.dev.jna jna 4.2.2 provided org.springframework spring-core 4.3.2.RELEASE compile org.springframework spring-web 4.3.2.RELEASE compile org.springframework.security spring-security-core 4.2.1.RELEASE provided org.apache.logging.log4j log4j-api 2.7 compile org.apache.logging.log4j log4j-core 2.7 compile org.meanbean meanbean 2.0.3 test nl.jqno.equalsverifier equalsverifier 1.7.7 test org.projectlombok lombok 1.14.8 compile com.google.guava guava 18.0 compile com.tango.common.utils tango-common-utils 1.1.0 compile com.tango.common.mvc tango-common-mvc 1.1.3-SNAPSHOT compile com.tango.ims ims-client 2.2.4-SNAPSHOT compile junit junit 4.12 test org.mockito mockito-core 1.10.19 test //John.

TimV · April 12, 2017, 11:28pm

Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: org.apache.commons.logging.impl.AvalonLogger
jar1: /usr/share/elasticsearch/plugins/x-pack/commons-logging-1.1.3.jar
jar2: /usr/share/elasticsearch/plugins/x-pack/extensions/.installing-1737085689021563353/ims-realm-1.0.0-SNAPSHOT.jar

Somewhere in your chain of dependencies you are including commons-logging and then repackaging it into your uberjar.

The best way to fixing that is going depend on what your build process looks like.

If you're using maven, then

 mvn dependency:tree

will show you all your dependencies and how they're being included. Then you could add an explicit exclusion on commons-logging.

Alternatively it might be easier to do it at the stage where you build the uberjar. Just drop any dependencies that are part of x-pack.

jhn134910 · April 13, 2017, 8:25am

Thank you Tim - I've followed your guidance and have successfully installed my custom realm.
My custom realm attempts to authenticate against an external system by sending a HTTP request to that system. On initialization it attempts to create the HTTP client. However, I'm receiving a permissions problem:

Caused by: java.security.AccessControlException: access denied ("java.net.NetPermission" "getProxySelector")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_92]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_92]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_92]
at java.net.ProxySelector.getDefault(ProxySelector.java:94) ~[?:1.8.0_92]
at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1147) ~[?:?]
at org.apache.http.impl.client.HttpClients.createSystem(HttpClients.java:66) ~[?:?]
at org.springframework.http.client.HttpComponentsClientHttpRequestFactory.(HttpComponentsClientHttpRequestFactory.java:88) ~[?:?]
at org.springframework.http.client.HttpComponentsAsyncClientHttpRequestFactory.(HttpComponentsAsyncClientHttpRequestFactory.java:67) ~[?:?]
at com.tango.common.mvc.client.AsyncClient.(AsyncClient.java:61) ~[?:?]
at com.tango.common.mvc.client.AsyncClient.(AsyncClient.java:45) ~[?:?]
at com.tango.common.mvc.client.AsyncClient$Builder.build(AsyncClient.java:267) ~[?:?]
at com.tango.xpack.realm.IMSRealm.(IMSRealm.java:67) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:19) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:11) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.doStart(Realms.java:82) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.1.2.jar:5.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_92]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_92]
at org.elasticsearch.node.Node.start(Node.java:542) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:234) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:308) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]

I attempted to add a plugin-security.policy file on my classpath with the following grant -
permission java.net.NetPermission "*";

However, I get the same result? Any ideas how to resolve this one?

jhn134910 · April 13, 2017, 2:24pm

I see somebody else had the same issue:

I followed the steps suggested there. Unfortunately, though it doesn't seem to be picking up the permission I specified in x-pack-extension-security.policy. I add code like:

if(sm != null) {
sm.checkPermission(new NetworkPermission("getProxySelector"));
}
AccessController.doPrivileged((PrivilegedAction) () -> {
final AsyncClient asyncClient = new AsyncClient.Builder()
.withConnectionTimeout(connectionTimeout).withSocketTimeout(socketTimeout)
.withMaxConnections(maxConnections).withMaxConnectionsPerHost(maxConnectionsPerHost).build();

this.imsClient = new IMSClient("http://localhost", Optional.of(asyncClient), imsCacheSize, imsCacheExpiryInMinutes);
return null;

});

However, the checkPermission is saying I don't have permission even after the I okayed the permissions when installing the custom realm plugin and the .policy file is present

[root@iel-dev-tfsr-vm1 tango-ims-realm]# pwd
/usr/share/elasticsearch/plugins/x-pack/extensions/tango-ims-realm
[root@iel-dev-tfsr-vm1 tango-ims-realm]# ls
ims-realm-1.0.0-SNAPSHOT.jar x-pack-extension-descriptor.properties x-pack-extension-security.policy

jaymode · April 13, 2017, 4:39pm

Can you share the full contents of the policy file? Is it the same exact exception?

jhn134910 · April 14, 2017, 8:00am

Thanks for your response.
The contents of my x-pack-extension-descriptor.properties:
description=Tango IMS Realm Extension
version=1.0.0-SNAPSHOT
name=tango-ims-realm
classname=com.tango.xpack.IMSRealmExtension
java.version=1.8
xpack.version=5.1.2

The contents of my x-pack-extension-security.policy:
grant {
permission java.net.NetPermission "getProxySelector";
};

I'm building my fat JAR with maven assembly plugin.
Then when attempting to install I do the following:
[root@iel-dev-tfsr-vm1 x-pack]# ./extension remove tango-ims-realm
-> Removing tango-ims-realm...
[root@iel-dev-tfsr-vm1 x-pack]# ./extension install file:///home/tango/john/tango-ims-realm-1.0.0-SNAPSHOT.zip
-> Downloading file:///home/tango/john/tango-ims-realm-1.0.0-SNAPSHOT.zip
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: x-pack extension requires additional permissions @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

java.net.NetPermission getProxySelector
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed tango-ims-realm
[root@iel-dev-tfsr-vm1 x-pack]# pwd
/usr/share/elasticsearch/bin/x-pack
[root@iel-dev-tfsr-vm1 x-pack]# cd ../../plugins/x-pack/extensions/;chmod 755 *
[root@iel-dev-tfsr-vm1 extensions]# !1667
chown -R elasticsearch:elasticsearch /usr/share/elasticsearch && cd /var/log/elasticsearch/ && tail -f tango-cluster.log
[2017-04-14T08:53:05,484][INFO ][o.e.n.Node ] [] initializing ...
[2017-04-14T08:53:05,554][INFO ][o.e.e.NodeEnvironment ] [vCUgnmd] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [2.8gb], net total_space [18.4gb], spins? [unknown], types [rootfs]
[2017-04-14T08:53:05,554][INFO ][o.e.e.NodeEnvironment ] [vCUgnmd] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-04-14T08:53:05,599][INFO ][o.e.n.Node ] node name [vCUgnmd] derived from node ID [vCUgnmdpTsevehNdEv9E5A]; set [node.name] to override
[2017-04-14T08:53:05,601][INFO ][o.e.n.Node ] version[5.1.2], pid[36275], build[c8c4c16/2017-01-11T20:18:39.146Z], OS[Linux/3.10.0-229.20.1.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_92/25.92-b14]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [aggs-matrix-stats]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [ingest-common]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-expression]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-groovy]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-mustache]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [lang-painless]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [percolator]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [reindex]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [transport-netty3]
[2017-04-14T08:53:07,077][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded module [transport-netty4]
[2017-04-14T08:53:07,078][INFO ][o.e.p.PluginsService ] [vCUgnmd] loaded plugin [x-pack]
[2017-04-14T08:53:09,624][INFO ][o.e.n.Node ] initialized
[2017-04-14T08:53:09,625][INFO ][o.e.n.Node ] [vCUgnmd] starting ...
[2017-04-14T08:53:09,628][ERROR][o.e.b.Bootstrap ] [vCUgnmd] Exception
...
org.elasticsearch.bootstrap.StartupException: java.security.AccessControlException: access denied ("jdk.net.NetworkPermission" "getProxySelector")
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:89) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:82) ~[elasticsearch-5.1.2.jar:5.1.2]
Caused by: java.security.AccessControlException: access denied ("jdk.net.NetworkPermission" "getProxySelector")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_92]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_92]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_92]
at com.tango.xpack.realm.IMSRealm.(IMSRealm.java:69) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:19) ~[?:?]
at com.tango.xpack.realm.IMSRealmFactory.create(IMSRealmFactory.java:11) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.initRealms(Realms.java:182) ~[?:?]
at org.elasticsearch.xpack.security.authc.Realms.doStart(Realms.java:82) ~[?:?]
at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.1.2.jar:5.1.2]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_92]
at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1080) ~[?:1.8.0_92]
at org.elasticsearch.node.Node.start(Node.java:542) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:234) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:308) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-5.1.2.jar:5.1.2]

jaymode · April 14, 2017, 11:21am

It looks like it might boil down to just an incorrect permission name:

Try using jdk.net.NetworkPermission instead of java.net.NetPermission in your policy file.

jhn134910 · April 14, 2017, 12:59pm

Jeez - I should have spotted that. Staring at it too long.
This mismatch was caused by my coding:

if(sm != null) {
sm.checkPermission(new NetworkPermission("getProxySelector"));
}

When I changed it to the correct

if(sm != null) {
sm.checkPermission(new NetPermission("getProxySelector"));
}

then the exception lines up with what I have in my grant file.

When I add the line
permission java.net.NetPermission "getProxySelector";

to the file:
/usr/share/elasticsearch/plugins/x-pack/plugin-security.policy

then is seems to work, but I guess this is giving the permission right across x-pack. Should my policy file also be called plugin-security.policy?

jaymode · April 14, 2017, 2:20pm

This is only granted to your extension.

No, x-pack-extension-security.policy is the correct name.

system · May 12, 2017, 2:34pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.