2.2.0 ESIntegTestCase - ClassNotFoundException when executing groovy script in search

Hi,
I've been using Elastic 1.4.4, but we're now upgrading to 2.2.0. I am having trouble getting my integration tests to run. The error I'm getting is following:

{
    "error": {
        "root_cause": [{
            "type": "script_exception",
            "reason": "failed to compile groovy script"
        }],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [{
            "shard": 0,
            "index": "bokun",
            "node": "BNyjts9hTOicRgCAWGdKgQ",
            "reason": {
                "type": "script_exception",
                "reason": "Failed to compile inline script [if(_source.accumulated_availability != null){  for(item in _source.accumulated_availability){    if(start.compareTo(item.day) < 0 && (end == null || end.compareTo(item.day) >= 0)){      return item.day    }  }} else return null;] using lang [groovy]",
                "caused_by": {
                    "type": "script_exception",
                    "reason": "failed to compile groovy script",
                    "caused_by": {
                        "type": "multiple_compilation_errors_exception",
                        "reason": "startup failed:\nCould not instantiate global transform class groovy.grape.GrabAnnotationTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.4-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation\n\nCould not instantiate global transform class org.codehaus.groovy.ast.builder.AstBuilderTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.4-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation\n\n2 errors\n"
                    }
                }
            }
        }]
    },
    "status": 500
}

What can cause this? I have the following dependencies in my build.sbt:

  "org.codehaus.groovy" % "groovy-all" % "2.4.4",
  "com.carrotsearch.randomizedtesting" % "randomizedtesting-runner" % "2.3.0" % "test",
  "org.apache.lucene" % "lucene-test-framework" % "5.4.1",
  "org.elasticsearch" % "elasticsearch" % "2.2.0" % "test" classifier "tests" withSources(),
  "org.elasticsearch" % "elasticsearch" % "2.2.0" withSources(),
  "org.elasticsearch.plugin" % "analysis-icu" % "2.2.0" % "test",
  "org.elasticsearch.module" % "lang-groovy" % "2.2.0" % "test"

...and I have the following in my EsIntegTestCase extension class:

@Override
protected Settings nodeSettings(int nodeOrdinal) {
    return Settings.settingsBuilder()
            .put(super.nodeSettings(nodeOrdinal))
            .put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, 1)
            .put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 1)
            .put(Node.HTTP_ENABLED, true)
            .put("script.groovy.sandbox.enabled", true)
            .put("script.engine.groovy.inline.search", true)
            .put("script.engine.groovy.inline.update", "true")
            .put("script.inline", true)
            .put("script.update", true)
            .put("script.indexed", true)
            .put("script.default_lang", "groovy")
            .build();
}

@Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
    return pluginList(GroovyPlugin.class, AnalysisICUPlugin.class);
}

I'm completely stuck, and Google is unwilling to help! :slightly_smiling:

Any help or pointers would be greatly appreciated.

Many thanks,
OGG

Hi OGG,

It looks like your problem is that you are importing groovy-all without bringing in the indy jar. I am not familiar with Scala's build system, but using the test jar as an example, you need to change your Groovy dependency to match ES:

 "org.codehaus.groovy" % "groovy-all" % "2.4.4" classifier "indy"

You can see the Groovy dependency used by Elasticsearch 2.2.0 here. Note that the latest version of Groovy is 2.4.6, which we will depend on in ES 2.3 and ES 5.0 (unless of course another release is made before we make those releases!).

Note that the Groovy artifact in future releases has been modified from groovy-all to just groovy.

1 Like

Hi and thanks for your reply, @pickypg.
I think it was using the "indy" jar before, if you look at the deepest "reason" in the error JSON I posted, it looks like this:

startup failed:

Could not instantiate global transform class 
groovy.grape.GrabAnnotationTransformation 
specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.4-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  

because of exception 
java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation

Could not instantiate global transform class 
org.codehaus.groovy.ast.builder.AstBuilderTransformation 
specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.4-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  

because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation

As you can see, it is referencing the groovy-all-2.4.4-indy.jar file.

However, to be absolutely sure, I modified my SBT dependency so that it now looks like this:

"org.codehaus.groovy" % "groovy-all" % "2.4.4" classifier "indy"

But unfortunately, I still get the exact same error back. :worried:

Without the indy classifier, then you were actually using both, which causes all sorts of problems when they try to use the same code. I'm actually quite surprised that that did not qualify as "jar hell". Perhaps the Scala Build Tool is better than I give it credit and it's actually deduping that.

You actually should be able to remove your Groovy dependency entirely since it's a transitive dependency of the module, but that shouldn't matter with the right classifier.

One of the more annoying/confusing aspects of Java is that a duplicated class usually leads to confusing ClassNotFoundExceptions because it tries to use classes across jars, which are effectively out of scope from each other. Worse: once a NoClassDefError (the true source of the exception) occurs, then there's no correcting it.

I still get the exact same error back.

Looking at the very end of your exception output:

org.codehaus.groovy.ast.builder.AstBuilderTransformation\n\n2 errors

This suggests that there was probably something that appeared in your logs. Can you check the logs around the error, then post the "2 errors"?

Thanks

Thanks for the reply, @pickypg. I think the "2 errors" just refers to the two errors listed before.

I tried switching to Groovy 2.4.6, but still get the same error.

{
    "error": {
        "root_cause": [{
            "type": "script_exception",
            "reason": "failed to compile groovy script"
        }],
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [{
            "shard": 0,
            "index": "bokun",
            "node": "rOAuO9p6Qem9F-WdBFrLXw",
            "reason": {
                "type": "script_exception",
                "reason": "Failed to compile inline script [if(_source.accumulated_availability != null){  for(item in _source.accumulated_availability){    if(start.compareTo(item.day) < 0 && (end == null || end.compareTo(item.day) >= 0)){      return item.day    }  }} else return null;] using lang [groovy]",
                "caused_by": {
                    "type": "script_exception",
                    "reason": "failed to compile groovy script",
                    "caused_by": {
                        "type": "multiple_compilation_errors_exception",
                        "reason": "startup failed:\nCould not instantiate global transform class groovy.grape.GrabAnnotationTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation\n\nCould not instantiate global transform class org.codehaus.groovy.ast.builder.AstBuilderTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation\n\n2 errors\n"
                    }
                }
            }
        }]
    },
    "status": 500
}

If I format the "reason" from the JSON above, I get:

startup failed:

Could not instantiate global transform class groovy.grape.GrabAnnotationTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  
because of exception java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation

Could not instantiate global transform class org.codehaus.groovy.ast.builder.AstBuilderTransformation specified at jar:file:/Users/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation

2 errors

My dependency is now:

"org.codehaus.groovy" % "groovy-all" % "2.4.6" classifier "indy"

Does the elastic search source code have an integration test for groovy script fields that I could look at? I did not find any over at Github.

Many thanks again for your help.

What happens if you run without the security manager enabled (pass it into the build!)?

-Dsecurity.manager.enabled=false

Does the Elasticsearch source code have an integration test for groovy script fields that I could look at? I did not find any over at Github.

They've been isolated under the Groovy module (module is just a fancy way to say pre-installed plugin):

https://github.com/elastic/elasticsearch/tree/master/modules/lang-groovy/src/test/java/org/elasticsearch/messy/tests

I'm actually already running the tests with the security manager disabled (sorry I did not mention that before):

activator -jvm-debug -J-ea -Dtests.timezone=UTC -Djava.awt.headless=true -Dtests.security.manager=false

If I don't do that, I immediately get an error:

[error] Test webservices.ActivityApiTest failed: java.security.AccessControlException: access denied ("org.elasticsearch.ThreadPermission" "modifyArbitraryThreadGroup"), took 0.0 sec
[error]     at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
[error]     at java.security.AccessController.checkPermission(AccessController.java:884)
[error]     at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[error]     at org.elasticsearch.SecureSM.checkThreadGroupAccess(SecureSM.java:166)
[error]     at org.elasticsearch.SecureSM.checkAccess(SecureSM.java:113)
[error]     at java.lang.ThreadGroup.checkAccess(ThreadGroup.java:315)
[error]     at java.lang.ThreadGroup.getParent(ThreadGroup.java:167)
[error]     at com.carrotsearch.randomizedtesting.Threads$2.run(Threads.java:127)
[error]     at com.carrotsearch.randomizedtesting.Threads$2.run(Threads.java:123)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at com.carrotsearch.randomizedtesting.Threads.getTopThreadGroup(Threads.java:123)
[error]     at com.carrotsearch.randomizedtesting.Threads.getAllThreads(Threads.java:99)
[error]     at com.carrotsearch.randomizedtesting.ThreadLeakControl.<init>(ThreadLeakControl.java:348)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:673)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
[error] Test null failed: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "setDefaultUncaughtExceptionHandler"), took 0.0 sec
[error]     at java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
[error]     at java.security.AccessController.checkPermission(AccessController.java:884)
[error]     at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
[error]     at java.lang.Thread.setDefaultUncaughtExceptionHandler(Thread.java:1893)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner$3.run(RandomizedRunner.java:618)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner$3.run(RandomizedRunner.java:615)
[error]     at java.security.AccessController.doPrivileged(Native Method)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:615)
[error]     at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)

Thanks for the link to the integration tests, I'm going to read through those.

@pickypg I copied one of the integration tests (https://github.com/elastic/elasticsearch/blob/master/modules/lang-groovy/src/test/java/org/elasticsearch/messy/tests/SearchFieldsTests.java) into my project and tried running it to see if I would get another result.

I get an error, but this time I do get a much more detailed stack trace. Does this help at all?

[error] Test srch.SearchFieldsTest.testScriptFieldUsingSource failed: org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed, took 0.479 sec
[error]     at __randomizedtesting.SeedInfo.seed([CEF40589697DFE61:F5614DE377F5610E]:0)
[error]     at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:228)
[error]     at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:174)
[error]     at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:46)
[error]     at org.elasticsearch.transport.local.LocalTransport.handleException(LocalTransport.java:354)
...
[error] Caused by: ScriptException[failed to compile groovy script]; nested: MultipleCompilationErrorsException[startup failed:
[error] Could not instantiate global transform class groovy.grape.GrabAnnotationTransformation specified at jar:file:/home/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation
[error] 
[error] Could not instantiate global transform class org.codehaus.groovy.ast.builder.AstBuilderTransformation specified at jar:file:/home/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation
[error] 
[error] 2 errors
[error] ];
[error]     at org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:196)
[error]     at org.elasticsearch.script.ScriptService.compileInternal(ScriptService.java:310)
[error]     at org.elasticsearch.script.ScriptService.compile(ScriptService.java:256)
[error]     at org.elasticsearch.script.ScriptService.search(ScriptService.java:455)
[error]     at org.elasticsearch.search.fetch.script.ScriptFieldsParseElement.parse(ScriptFieldsParseElement.java:101)
[error]     at org.elasticsearch.search.SearchService.parseSource(SearchService.java:836)
[error]     at org.elasticsearch.search.SearchService.createContext(SearchService.java:652)
...
[error]     ... 3 more
[error] Caused by: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
[error] Could not instantiate global transform class groovy.grape.GrabAnnotationTransformation specified at jar:file:/home/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: groovy.grape.GrabAnnotationTransformation
[error] 
[error] Could not instantiate global transform class org.codehaus.groovy.ast.builder.AstBuilderTransformation specified at jar:file:/home/ogg/.ivy2/cache/org.codehaus.groovy/groovy-all/jars/groovy-all-2.4.6-indy.jar!/META-INF/services/org.codehaus.groovy.transform.ASTTransformation  because of exception java.lang.ClassNotFoundException: org.codehaus.groovy.ast.builder.AstBuilderTransformation
[error] 
[error] 2 errors
[error]     at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
[error]     at org.codehaus.groovy.control.ProcessingUnit.completePhase(ProcessingUnit.java:144)
[error]     at org.codehaus.groovy.control.ProcessingUnit.gotoPhase(ProcessingUnit.java:163)
[error]     at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:536)
[error]     at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
[error]     at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
[error]     at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
[error]     at org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:189)
[error]     at org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:183)
[error]     at java.security.AccessController.doPrivileged(Native Method)

Hi,
I fired up my debugger, and it looks like this is actually a SecurityException which is being rethrown as ClassNotFoundException. The following snippet throws the exception (this is from org.elasticsearch.script.groovy.GroovyScriptEngineService):

this.loader = AccessController.doPrivileged(new PrivilegedAction<GroovyClassLoader>() {
    @Override
    public GroovyClassLoader run() {
        // snapshot our context (which has permissions for classes), since the script has none
        final AccessControlContext engineContext = AccessController.getContext();
        return new GroovyClassLoader(new ClassLoader(getClass().getClassLoader()) {
            @Override
            protected Class<?> loadClass(String name, boolean resolve) throws ClassNotFoundException {
                if (sm != null) {
                    try {
                        engineContext.checkPermission(new ClassPermission(name));
                    } catch (SecurityException e) {
                        throw new ClassNotFoundException(name, e);
                    }
                }
                return super.loadClass(name, resolve);
            }
        }, config);
    }
});

The actual stack trace is:

java.security.AccessControlException: access denied ("org.elasticsearch.script.ClassPermission" "groovy.grape.GrabAnnotationTransformation")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
	at org.elasticsearch.script.groovy.GroovyScriptEngineService$1$1.loadClass(GroovyScriptEngineService.java:115)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addPhaseOperationsForGlobalTransforms(ASTTransformationVisitor.java:301)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.doAddGlobalTransforms(ASTTransformationVisitor.java:282)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addGlobalTransforms(ASTTransformationVisitor.java:192)
	at org.codehaus.groovy.transform.ASTTransformationVisitor.addPhaseOperations(ASTTransformationVisitor.java:156)
	at org.codehaus.groovy.control.CompilationUnit.<init>(CompilationUnit.java:206)
	at org.codehaus.groovy.control.CompilationUnit.<init>(CompilationUnit.java:123)
	at groovy.lang.GroovyClassLoader.createCompilationUnit(GroovyClassLoader.java:456)
	at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:277)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
	at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
	at org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:189)

Now the question is, how do I modify these class permissions?

I have managed to solve this.

Using the instructions at https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-security.html I created a security policy file:

grant {
    permission org.elasticsearch.script.ClassPermission "java.lang.Class";
    permission org.elasticsearch.script.ClassPermission "org.codehaus.groovy.*";
    permission org.elasticsearch.script.ClassPermission "groovy.*";
    permission org.elasticsearch.script.ClassPermission "java.lang.*";
    permission org.elasticsearch.script.ClassPermission "java.util.*";
    permission org.elasticsearch.script.ClassPermission "java.math.BigDecimal";
    permission org.elasticsearch.script.ClassPermission "org.joda.time.*";
};

And then I start the tests passing my security policy file on the command line:

-Djava.security.policy=security.policy

The groovy script fields are now executed without problems.

Thanks for your help.