ESTestCase RuntimeException when running with Gradle- jar hell in test classpath

dominik.safaric · February 14, 2018, 11:14am

I've been trying to run my ElasticSearch plugin unit tests (extending the ESTestcase class) using the following command ./gradlew, but unfortunately I keep getting the following exception:

Gradle Test Executor 1 finished executing tests.

org.elasticsearch.index.query.SimilaritySearchQueryBuilderTests > classMethod FAILED
    java.lang.RuntimeException: found jar hell in test classpath

        Caused by:
        java.nio.file.NoSuchFileException: /Developer/sds/image-search/build/resources/test
Exception in thread "Thread-4" java.lang.NoClassDefFoundError: Could not initialize class org.elasticsearch.test.ESTestCase
        at java.lang.Thread.run(Thread.java:745)
        Suppressed: java.lang.IllegalStateException: No context information for thread: Thread[id=14, name=Thread-4, state=RUNNABLE, group=TGRP-SimilaritySearchQueryBuilderTests]. Is this thread running under a class com.carrotsearch.randomizedtesting.RandomizedRunner runner context? Add @RunWith(class com.carrotsearch.randomizedtesting.RandomizedRunner.class) to your test class. Make sure your code accesses random contexts within @BeforeClass and @AfterClass boundary (for example, static test class initializers are not permitted to access random contexts).
                at com.carrotsearch.randomizedtesting.RandomizedContext.context(RandomizedContext.java:248)
                at com.carrotsearch.randomizedtesting.RandomizedContext.current(RandomizedContext.java:134)
                at com.carrotsearch.randomizedtesting.RandomizedRunner.augmentStackTrace(RandomizedRunner.java:1848)
                at com.carrotsearch.randomizedtesting.RunnerThreadGroup.uncaughtException(RunnerThreadGroup.java:20)
                at java.lang.Thread.dispatchUncaughtException(Thread.java:1952)

Interestedly, if I run the unit tests from the IDEA, the tests run without any problems, i.e. no jar hell related exception occurs. Following advice from the previous related discussions, I've excluded hamcrest-core from the junit dependency, but unfortunately none helper. Here you may find the build.gradle file of mine.

group 'group'
version '1.0-SNAPSHOT'

apply plugin: 'java'

sourceCompatibility = 1.8
targetCompatibility = 1.8

repositories {
    mavenCentral()
}

dependencies {
    compileOnly group: 'org.elasticsearch', name: 'elasticsearch', version: '5.3.2'

    testCompile group: 'org.apache.lucene', name: 'lucene-test-framework', version: '6.4.2'
    testCompile group: 'org.elasticsearch.test', name: 'framework', version: '5.3.2'
    testCompile group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.10.0'
    testCompile group: 'com.vividsolutions', name: 'jts', version: '1.13'
    testCompile group: 'org.locationtech.spatial4j', name: 'spatial4j', version: '0.6'
    testCompile (group: 'junit', name: 'junit', version: '4.12') {
        exclude module : 'hamcrest'
        exclude module : 'hamcrest-core'
    }
}

test {
    systemProperty 'tests.security.manager', System.getProperty('tests.security.manager')
}

Could please someone provide a hint onto why this issue happens when running the tests using Gradle?

rjernst · February 14, 2018, 9:51pm

Have you tried using the gradle plugin provided by Elastic to build ES plugins? This automatically sets up a test task which uses the ES test framework.

Aside from that, I see a couple things likely to cause problems. One is using compileOnly here. While that is correct, you need to make your testCompile extend from it (gradle does not do this). Something like this should work:

configurations.testCompile.extendsFrom configurations.compileOnly

Your issue is likely how the test is run. We don't use the gradle test runner, but instead the same test runner that Lucene uses: randomized testing framework. In order to use it, you need to have your tests either extend RandomizedTest, or annotate your tests to use the RandomizedRunner (see https://github.com/randomizedtesting/randomizedtesting/wiki/Randomized-Runner).

But even better would be to use the gradle plugin. Unfortunately we don't have any guide for this yet, but a simple build.gradle using it should look something like this:

buildscript {
    repositories {
        mavenCentral()
        jcenter()
    }
    dependencies {
        classpath "org.elasticsearch.gradle:build-tools:$elasticsearchVersion"
    }
}

apply plugin: 'elasticsearch.esplugin'

esplugin {
    name 'my-plugin-name'
    description 'a description of my plugin'
    classname 'pkg.MyClassExtendingPlugin'
}

dominik.safaric · February 19, 2018, 1:57pm

Could you please provide a more complete example that includes references to other dependencies, including compile_only dependencies such aslucene, elasticsearch and carrotsearch.randomized-testing? Asking because I'm having some trouble integrating the plugin:


* What went wrong:
A problem occurred evaluating root project 'similarity-search'.
> Failed to apply plugin [id 'carrotsearch.randomized-testing']
   > Could not create task of type 'RandomizedTestingTask'.

Furthermore, does the Gradle plugin generate the plugin-descriptor.properties file, or do I still have to define it separately in the build script?

rjernst · February 19, 2018, 5:34pm

The gradle plugin provides all of those dependencies automatically. Can you please run your gradle task with -s so we can see what is failing in RandomizedTestingTask initialization?

dominik.safaric · February 20, 2018, 2:03pm

This is the stacktrace of the exception:

Caused by: org.gradle.api.internal.plugins.PluginApplicationException: Failed to apply plugin [id 'carrotsearch.randomized-testing']
        at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:156)
        at org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:113)
        at org.gradle.api.plugins.PluginManager$apply$0.call(Unknown Source)
        at org.elasticsearch.gradle.BuildPlugin.apply(BuildPlugin.groovy:65)
        at org.elasticsearch.gradle.plugin.PluginBuildPlugin.super$2$apply(PluginBuildPlugin.groovy)
        at org.elasticsearch.gradle.plugin.PluginBuildPlugin.apply(PluginBuildPlugin.groovy:46)
        at org.elasticsearch.gradle.plugin.PluginBuildPlugin.apply(PluginBuildPlugin.groovy)
        at org.gradle.api.internal.plugins.ImperativeOnlyPluginApplicator.applyImperative(ImperativeOnlyPluginApplicator.java:35)
        at org.gradle.api.internal.plugins.RuleBasedPluginApplicator.applyImperative(RuleBasedPluginApplicator.java:43)
        at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:140)
        at org.gradle.api.internal.plugins.DefaultPluginManager.apply(DefaultPluginManager.java:113)
        at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.applyType(DefaultObjectConfigurationAction.java:113)
        at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.access$200(DefaultObjectConfigurationAction.java:36)
        at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction$3.run(DefaultObjectConfigurationAction.java:80)
        at org.gradle.api.internal.plugins.DefaultObjectConfigurationAction.execute(DefaultObjectConfigurationAction.java:136)
        at org.gradle.api.internal.project.AbstractPluginAware.apply(AbstractPluginAware.java:44)
        at org.gradle.api.internal.project.ProjectScript.apply(ProjectScript.java:34)
        at org.gradle.api.Script$apply$0.callCurrent(Unknown Source)
        at build_a2yuqjkmya3mh6577tcr4vohz.run(/home/dominiksafaric/Developer/org.styria/sds/image-search-lucene/build.gradle:11)
        at org.gradle.groovy.scripts.internal.DefaultScriptRunnerFactory$ScriptRunnerImpl.run(DefaultScriptRunnerFactory.java:90)
        ... 54 more
Caused by: org.gradle.api.tasks.TaskInstantiationException: Could not create task of type 'RandomizedTestingTask'.
        at org.gradle.api.internal.project.taskfactory.TaskFactory$1.call(TaskFactory.java:123)
        at org.gradle.api.internal.project.taskfactory.TaskFactory$1.call(TaskFactory.java:118)
        at org.gradle.util.GUtil.uncheckedCall(GUtil.java:402)
        at org.gradle.api.internal.AbstractTask.injectIntoNewInstance(AbstractTask.java:179)
        at org.gradle.api.internal.project.taskfactory.TaskFactory.create(TaskFactory.java:118)
        at org.gradle.api.internal.project.taskfactory.TaskFactory.createTask(TaskFactory.java:77)
        at org.gradle.api.internal.project.taskfactory.AnnotationProcessingTaskFactory.createTask(AnnotationProcessingTaskFactory.java:46)
        at org.gradle.api.internal.project.taskfactory.DependencyAutoWireTaskFactory.createTask(DependencyAutoWireTaskFactory.java:39)
        at org.gradle.api.internal.tasks.DefaultTaskContainer.create(DefaultTaskContainer.java:76)
        at org.gradle.api.tasks.TaskContainer$create$0.call(Unknown Source)
        at com.carrotsearch.gradle.junit4.RandomizedTestingPlugin.replaceTestTask(RandomizedTestingPlugin.groovy:34)
        at com.carrotsearch.gradle.junit4.RandomizedTestingPlugin$replaceTestTask.callStatic(Unknown Source)
        at com.carrotsearch.gradle.junit4.RandomizedTestingPlugin.apply(RandomizedTestingPlugin.groovy:15)
        at com.carrotsearch.gradle.junit4.RandomizedTestingPlugin.apply(RandomizedTestingPlugin.groovy)
        at org.gradle.api.internal.plugins.ImperativeOnlyPluginApplicator.applyImperative(ImperativeOnlyPluginApplicator.java:35)
        at org.gradle.api.internal.plugins.RuleBasedPluginApplicator.applyImperative(RuleBasedPluginApplicator.java:43)
        at org.gradle.api.internal.plugins.DefaultPluginManager.doApply(DefaultPluginManager.java:140)
        ... 73 more
Caused by: org.gradle.internal.service.UnknownServiceException: No service of type ProgressLoggerFactory available in ProjectScopeServices.
        at org.gradle.internal.service.DefaultServiceRegistry.getServiceProvider(DefaultServiceRegistry.java:436)
        at org.gradle.internal.service.DefaultServiceRegistry.doGet(DefaultServiceRegistry.java:426)
        at org.gradle.internal.service.DefaultServiceRegistry.get(DefaultServiceRegistry.java:414)
        at com.carrotsearch.gradle.junit4.RandomizedTestingTask_Decorated.getProgressLoggerFactory(Unknown Source)
        at com.carrotsearch.gradle.junit4.RandomizedTestingTask.<init>(RandomizedTestingTask.groovy:87)
        at com.carrotsearch.gradle.junit4.RandomizedTestingTask_Decorated.<init>(Unknown Source)
        at org.gradle.api.internal.DependencyInjectingInstantiator.newInstance(DependencyInjectingInstantiator.java:56)
        at org.gradle.api.internal.ClassGeneratorBackedInstantiator.newInstance(ClassGeneratorBackedInstantiator.java:36)
        at org.gradle.api.internal.project.taskfactory.TaskFactory$1.call(TaskFactory.java:121)
        ... 89 more

I guess the problem is due to the ProgressLoggerFactory class Gradle guys decided to drop out.

FYI, the Gradle version I'm using is 3.5-rc2, whereas the ElasticSearch version is 5.3.2.

rjernst · February 21, 2018, 9:54pm

Gradle moved stuff around and made their progress logging functionality "internal" to gradle. We still use it because without it the output would be horrible to look at (pages of printlns). IIRC, for Elasticsearch 5.3 you will need to use gradle 2.13 or 2.14 (I believe either will work).

rjernst · February 21, 2018, 9:55pm

Note that in recent versions (6.2+, and 5.6 branch), we have begun using the gradle wrapper, so you no longer need to worry about which version of gradle the build is compatible with.

dominik.safaric · February 22, 2018, 11:21am

This is the information I sought for. By changing Gradle's distributionUrl to 2.13 everything works perfectly fine. Thanks a lot

jasontedor · February 22, 2018, 11:34am

By the way, we have backported the Gradle wrapper to all branches in 5.x and 6.x as well so you can use ./gradlew from the root of the repository and see:

06:34:12 [jason:~/src/elastic/elasticsearch-5.x] 5.3+ ± JAVA_HOME=`/usr/libexec/java_home -v 1.8` ./gradlew --version

------------------------------------------------------------
Gradle 2.13
------------------------------------------------------------

Build time:   2016-04-25 04:10:10 UTC
Build number: none
Revision:     3b427b1481e46232107303c90be7b05079b05b1c

Groovy:       2.4.4
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_161 (Oracle Corporation 25.161-b12)
OS:           Mac OS X 10.13.3 x86_64

system · March 22, 2018, 11:35am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.