Cutom Realm build JarHell with Guava

[jhn134910 (John)]

Hello there,
I'm using elastic-search v6.2.3, with x-pack plugin-in v6.2.3 installed.
The deployment used RPMs and my OS is centos 7. I've conducted a recent upgrade from 5.4.0 to 5.6.8 to 6.2.3.
As part of the upgrade work I needed to upgrade my custom x-pack authentication realm which was working for 5.4.0.

My x-pack plugin depends on google guava:

 [jlee@iel-jlee-lt1 tango-xpack-ims-realm]$ mvn dependency:tree -Dverbose | grep guava
[INFO] +- com.google.guava:guava:jar:19.0:compile

When I attempt to install my custom realm plugin:

 [tango@iel-dev-mtn-vm2 ~]$ sudo /usr/share/elasticsearch/bin/x-pack/extension install file:///home/tango/tarballs/tango-elasticsearch/security/tango-ims-realm-1.2.1-SNAPSHOT.zip
-> Downloading file:///home/tango/tarballs/tango-elasticsearch/security/tango-ims-realm-1.2.1-SNAPSHOT.zip
Exception in thread "main" java.lang.IllegalStateException: jar hell!
class: com.google.common.annotations.Beta
jar1: /usr/share/elasticsearch/plugins/x-pack/x-pack-security/guava-19.0.jar
jar2: /usr/share/elasticsearch/plugins/x-pack/x-pack-security/extensions/.installing-2586954945394098920/tango-ims-realm-1.2.1-SNAPSHOT.jar
	at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:275)
	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:192)
	at org.elasticsearch.xpack.core.extensions.InstallXPackExtensionCommand.jarHellCheck(InstallXPackExtensionCommand.java:196)
..
	at org.elasticsearch.xpack.core.extensions.XPackExtensionCli.main(XPackExtensionCli.java:37)

My understanding is that there is no dependency in either elastic-search or x-pack on guava:
https://github.com/elastic/elasticsearch/issues/13224.

I was hoping to avoid shading (https://www.elastic.co/blog/to-shade-or-not-to-shade) if possible.
Any ideas how to get to the bottom of this one?

[jhn134910 (John)]

Actually, I think guava and sl4fj-api are placed on the classpath as part of the x-pack installation at:

[tango@iel-dev-mtn-vm2 x-pack-security]$ pwd
/usr/share/elasticsearch/plugins/x-pack/x-pack-security

[tango@iel-dev-mtn-vm2 x-pack-security]$ ll
total 8192
-rw-r--r-- 1 elasticsearch elasticsearch  150171 Apr  5 16:04 cryptacular-1.2.0.jar
drwxr-xr-x 2 elasticsearch elasticsearch       6 Apr  9 14:43 extensions
-rw-r--r-- 1 elasticsearch elasticsearch 2308517 Apr  5 16:04 guava-19.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  158984 Apr  5 16:04 httpclient-cache-4.5.2.jar
-rw-r--r-- 1 elasticsearch elasticsearch  233305 Apr  5 16:04 java-support-7.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   49465 Apr  5 16:04 LICENSE.txt
-rw-r--r-- 1 elasticsearch elasticsearch   23153 Apr  5 16:04 log4j-slf4j-impl-2.9.1.jar
-rw-r--r-- 1 elasticsearch elasticsearch  129402 Apr  5 16:04 metrics-core-3.2.2.jar
-rw-r--r-- 1 elasticsearch elasticsearch  152158 Apr  5 16:04 NOTICE.txt
-rw-r--r-- 1 elasticsearch elasticsearch  177928 Apr  5 16:04 opensaml-core-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   53160 Apr  5 16:04 opensaml-messaging-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   12603 Apr  5 16:04 opensaml-messaging-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   27040 Apr  5 16:04 opensaml-profile-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   28145 Apr  5 16:04 opensaml-profile-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  397385 Apr  5 16:04 opensaml-saml-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch 1128664 Apr  5 16:04 opensaml-saml-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   60830 Apr  5 16:04 opensaml-security-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   97689 Apr  5 16:04 opensaml-security-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  160741 Apr  5 16:04 opensaml-soap-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  403093 Apr  5 16:04 opensaml-soap-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   23544 Apr  5 16:04 opensaml-storage-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch   76799 Apr  5 16:04 opensaml-storage-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  206288 Apr  5 16:04 opensaml-xmlsec-api-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch  292460 Apr  5 16:04 opensaml-xmlsec-impl-3.3.0.jar
-rw-r--r-- 1 elasticsearch elasticsearch    1649 Apr  5 16:04 plugin-descriptor.properties
-rw-r--r-- 1 elasticsearch elasticsearch    2868 Apr  5 16:04 plugin-security.policy
-rw-r--r-- 1 elasticsearch elasticsearch   25689 Apr  5 16:04 slf4j-api-1.6.2.jar
-rw-r--r-- 1 elasticsearch elasticsearch 1133022 Apr  5 16:04 xmlsec-2.0.8.jar
-rw-r--r-- 1 elasticsearch elasticsearch  821311 Apr  5 16:04 x-pack-security-6.2.3.jar

so even though guava removed by elastic-search is is still used by x-pack? I'm still confused about how to proceed however?

[TimV (Tim Vernum)]

In X-Pack 6.2 we made a number of changes that affect the way X-Pack is packaged, and also introduced a number of new jar dependencies.

X-Pack security does include Guava (it's a transitive dependency from other libraries that we rely on) and that prevents you from including Guava in your own extension.
What makes that harder to deal with, is the fact that Guava is not compatible between major versions, so your extension can only use Guava 19.0

If you can live with that (and it appeas that you can, since that's in your dependency tree) then just mark the dependency as "provided" in your POM (or equivalent for other build systems) and it should all work fine.

We will be making a significant change to the packaging of extensions in 6.3, so this advice is only relevant for 6.2.x

[jhn134910 (John)]

Thanks for your reply Tim.
My code uses guava 19.0 as already mentioned. However, when I set the scope to provided when elastic-search starts it complains that it cannot find a particular guava class on the class path. I'm asserting that JARs listed above (in /usr/share/elasticsearch/plugins/x-pack/x-pack-security) are not actually on the elastic-search classpath?
To workaround my JarHell issue, I temporarily moved the guava-19.0.jar, installed my custom realm x-pack plugin successfully pulling in a compile time dependency on guava-19.0.jar, and then I restored guava-19.0.jar back on the file-system (/usr/share/elasticsearch/plugins/x-pack/x-pack-security). Then when I install elastic-search everything I need is on the classpath.
Looking at the JarHell class, the checkClass method could perhaps be more sophisticated and check the version of the JAR. As long as I'm using the same version of guava I guess there should be no problems and perhaps it shouldn't throw an exception when the versions of the JARs are the same?
Anyway, with this hack I get a little further even though it still isn't working as it seems I need to adjust my security permissions:

[2018-04-10T13:01:17,684][DEBUG][o.a.h.i.n.c.PoolingNHttpClientConnectionManager] Connection request: [route: {}->http://192.168.1.168:8225][total kept alive: 0; route allocated: 0 of 5; total allocated: 0 of 5]
[2018-04-10T13:01:17,704][ERROR][o.a.h.i.n.c.InternalHttpAsyncClient] I/O reactor terminated abnormally
java.security.AccessControlException: access denied ("java.net.SocketPermission" "192.168.1.168:8225" "connect,resolve")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_161]
	at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_161]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_161]
	at java.lang.SecurityManager.checkConnect(SecurityManager.java:1051) ~[?:1.8.0_161]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:625) ~[?:?]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processSessionRequests(DefaultConnectingIOReactor.java:273) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:139) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:348) ~[httpcore-nio-4.4.5.jar:4.4.5]
	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192) ~[httpasyncclient-4.1.2.jar:4.1.2]
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) [httpasyncclient-4.1.2.jar:4.1.2]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]