ElasticSearch 2.0 java api JarHell problems

George · November 17, 2015, 9:42am

Hello everybody,

I'm trying to migrate from elasticsearch 1.6.0 to 2.0.0 using the Java API. The tests were passing with 1.60. However, after I made the tests compile with 2.0.0 my tests are failing when running them locally due to various exceptions generated by JarHell. The question is, can I disable the JarHell check for local testing?

Thanks!

dadoonet · November 17, 2015, 9:54am

If you are running your tests from IntelliJ, that can be the cause.
Otherwise, I would try to fix any JAR conflict in my project instead of trying to disable JarHell check.

George · November 17, 2015, 10:28am

Hello,

Thanks for the response.
Actually , I'm running the tests using gradle 2.6, jdk 7 also tried it with 8.
Could you please tell me how could I work around such an issue ?

java.lang.RuntimeException: found jar hell in test classpath
Caused by:
java.lang.IllegalStateException: jar hell!
class: org.apache.hadoop.yarn.factories.package-info
jar1: /home/george/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-yarn-api/2.5.0-cdh5.3.0/7542d2b9b89956291d8fb4294aab143adf4f5252/hadoop-yarn-api-2.5.0-cdh5.3.0.jar
jar2: /home/george/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-yarn-common/2.5.0-cdh5.3.0/ab28ad57af245463f66a5da314dffc684fe9cc13/hadoop-yarn-common-2.5.0-cdh5.3.0.jar

This is about the package-info that happens to live in the same package in two different jars.

Thanks!

jprante · November 17, 2015, 11:00am

The "identical jar" is not a jar hell by Oracle's definition for a class path, but it was refused to fix: https://github.com/elastic/elasticsearch/issues/13604

So you have to do some very weird acrobatics in a gradle build, something like this for integration testing

task integrationTest(type: Test, dependsOn: ['unpackPlugin']) {
    testClassesDir = sourceSets.integrationTest.output.classesDir
    // the test dependencies including Elasticsearch jars
    classpath = configurations.integrationTestCompile
    // the dependencies of this plugin as jars from the plugin zip
    classpath += fileTree('plugins/<name>').include('*.jar')
    // our plugin classes and resources
    classpath += sourceSets.integrationTest.output
    // without removing identical jars from classpath, Elasticsearch whines about a "jar hell"
    classpath -= configurations.releaseJars
    outputs.upToDateWhen { false }
    systemProperty 'path.home', projectDir.absolutePath
}

The complete gradle build file is at https://github.com/jprante/elasticsearch-webapp/blob/master/build.gradle
It's a groovy plugin, but the 'identical jar' challenge also applies to Java plugin builds with gradle.

George · November 17, 2015, 11:27am

Hello,

But to be honest this JarHell is really restrictive. I mean take for example this package-info, which is actually not used at runtime. Why would you want to fail the build because of that? I should be able to decide as a client if this is fine with me or not. Elasticsearch should not decide about incompatibilities in other libraries in the same classpath, in my opinion.

@jprante Can you please suggest some way to run the tests locally and avoid JarHell? I have a simple test with no plugins, no nothing, only extending the ESIntegTestCase creating a mapping and retrievering it. I'm using the hadoop dependecy because I have a custom definition of the fields. Because I see that your gradle integrationTest depends on some unpackPlugin, which is not an option for me.

jprante · November 17, 2015, 11:49am

@George I never tried ESIntegTestCase, it looks too tricky to me

dadoonet · November 17, 2015, 12:36pm

I see. I tried also once to play with Hadoop but it's really a JAR hell there!

Not sure about what you can do. I think @costin solved that issue on his end for the hadoop plugin. @costin how did you fix that?

costin · November 17, 2015, 12:52pm

Load Hadoop into its own ClassLoader so that it's not part of the ES/JarHell classpath.

George · November 17, 2015, 1:09pm

Hi Costin,

Thanks for the response.
Can I do that in gradle when I run the test? Do you have some snippet or some example somewhere?

@dadoonet thanks for asking costin.
Thanks!

costin · November 17, 2015, 1:37pm

Not that I'm aware of - it's doable probably through some type of plugin however note that the JarHell can occur easily at runtime as well.
You can see the ClassLoader solution (which is Java Security Manager friendly as it does not require extra permissions) is here, in the repository-hdfs repo

anjith.p · January 15, 2016, 5:47am

I'm getting the same problem while running tests from Eclipse - running from maven works. Is there any solution to fix this issue in Eclipse ?

Caused by: java.lang.IllegalStateException: jar hell!
class: com...agent.restapi.transferobject.ClientRequestPopto
jar1: C:\WorkSoft\mvn-repo-security\com**\2.5.00-RELEASE*-2.5.00-RELEASE.jar
jar2: C:\p4ws\ola***\target\classes
at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:267)

dadoonet · January 15, 2016, 6:28am

Fix your classpath. You can't have the same class twice in the classloader.

anjith.p · January 15, 2016, 6:39am

@dadoonet, I have checked the classpath - generate by m2e plugin- and it doesn't have any duplicate entries - classpath is generated my maven eclipse integration. And, I'm gettng this problem only while running tests from the Eclipse. Any clues on what is going wrong ?

dadoonet · January 15, 2016, 7:20am

Can you print the full log?

anjith.p · January 15, 2016, 7:39am

Here is the trace:

TestBWElasticSearchSink.testProcess_BulkRequestWithChunks1
Unrooted Tests
com.hidden.ola.collection.payload.TestBWElasticSearchSink
java.lang.IllegalStateException: failed to load bundle [] due to jar hell

	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:342)

	at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:113)

	at org.elasticsearch.node.Node.<init>(Node.java:144)

	at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:145)

	at org.elasticsearch.node.NodeBuilder.node(NodeBuilder.java:152)

	at com.hidden.ola.util.ElasticSearchTest.setUpBeforeClass(ElasticSearchTest.java:79)

	at com.hidden.ola.collection.payload.TestBWElasticSearchSink.setUpBeforeClass(TestBWElasticSearchSink.java:46)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:497)

	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)

	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)

	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

Caused by: java.lang.IllegalStateException: jar hell!

class: com.hidden.ola.agent.restapi.transferobject.ClientRequestPopto

jar1: C:\WorkSoft\mvn-repo-security\com\hidden\ola\types\2.5.00-RELEASE\types-2.5.00-RELEASE.jar

jar2: C:\p4ws\ola\types\target\classes

	at org.elasticsearch.bootstrap.JarHell.checkClass(JarHell.java:267)

	at org.elasticsearch.bootstrap.JarHell$1.visitFile(JarHell.java:201)

	at org.elasticsearch.bootstrap.JarHell$1.visitFile(JarHell.java:194)

	at java.nio.file.Files.walkFileTree(Files.java:2670)

	at java.nio.file.Files.walkFileTree(Files.java:2742)

	at org.elasticsearch.bootstrap.JarHell.checkJarHell(JarHell.java:194)

	at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:340)

	... 22 more



java.lang.NullPointerException

	at com.hidden.ola.util.ElasticSearchTest.tearDownAfterClass(ElasticSearchTest.java:95)

	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

	at java.lang.reflect.Method.invoke(Method.java:497)

	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)

	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)

	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)

	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)

	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)

	at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)

	at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)

	at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

dadoonet · January 15, 2016, 7:59am

So this class com.hidden.ola.agent.restapi.transferobject.ClientRequestPopto is in two places in your classloader:

C:\WorkSoft\mvn-repo-security\com\hidden\ola\types\2.5.00-RELEASE\types-2.5.00-RELEASE.jar
C:\p4ws\ola\types\target\classes

May be remove the dependency to types-2.5.00-RELEASE?

anjith.p · January 15, 2016, 8:32am

That's what the log says, however, the same tests were successful when I ran them using maven - not sure if Eclipse is doing something crazy. I can't remove the types dependency: it is required. Is there anyway to disable the jar hell check ?

George · January 15, 2016, 8:50am

Hi Anjith,

You have to inspect the eclipse classpath carefully. Maybe you have some other open project and the class is loaded from there. If that is not the case, generate the eclipse artifacts using maven "mvn eclipse:eclipse" and inspect the .classpath.

Cheers,
George

anjith.p · January 15, 2016, 11:04am

Thanks all. We gave up on this issue. There are many class conflicts from 3rd party jars and it would take a lot of time to resolve all of those. We got the "jar hell" problem only while running unit tests; so we have decided to use external elastic search cluster for tests.

anjith.p · January 17, 2016, 5:24am

If any one gets into this problem while running unit tests, simply manipulate the java class path such that the class path contains only elasticsearch and lucene jars (don't forget to set the original class path once then es node has been started). This trick worked for me - agree that it's a hack.

String originalClassPath = System.getProperty("java.class.path");
String classPathEntries = originalClassPath.split(";");
StringBuilder esClasspath = new StringBuilder();
for (String entry : classPathEntries) {
if (entry.contains("elasticsearch") || entry.contains("lucene")) {
esClasspath.append(entry);
esClasspath.append(";");
}
}
System.setProperty("java.class.path", esClasspath.toString());
node = nodeBuilder().local(true).settings(settings).node();
System.setProperty("java.class.path", originalClassPath);