ElasticSearch Plugin Upgrade [1.7 to 5.4] -- Integration Testing

I'm currently working on upgrading a plugin to be compatible with our new 5.4 cluster. I've been following this tutorial, which has been very helpful, however I've hit a dead end in my quest for resolution...

Here's my setup:

MyPlugin (extends Plugin and implements ScriptPlugin) - this is the main plugin class which gathers all the factories to generate new scripts. I have not seen any tutorials saying there needs to be more in this class. Score, Rating and Tag all extend AbstractSearchScript, and their respective factories implement NativeScriptFactory

public List<NativeScriptFactory> getNativeScripts() {
    return Arrays.asList(new Score.Factory(),
            new Rating.Factory(),
            new Tag.Factory());

I followed all of the instructions, added the necessary Maven dependencies and implemented the ANT script from the ES5 Plugin Integration Test tutorial to get everything set and ready for testing. When I run mvn install, the test cluster spins up, and runs all of the Integration Tests I created for the scripts above. Here's an example of one of the tests:

public void runNegativeScore() throws Exception {

    //Adds a non-random doc to the ES Cluster customized for this test
    addImpactScoreStory(true, true, false);

    //Generates script parameters
    Map<String, Object> params = generateParameters(false, false);

    // Retrieve first 10 prime records
    SearchResponse searchResponse = client().prepareSearch("test")
            .addScriptField("scores", new Script(ScriptType.FILE, "native", "Score", params))
            //  .addSort("number", SortOrder.ASC)

    SearchHits hits = searchResponse.getHits();
    for (SearchHit hit : hits) {
        if (hit.getId().equals("1")) {
            double[] values = hit.getField("scores").getValue();
            assert (Math.round(values[0] * 100) / 100d == -.22);
            assert (Math.round(values[1] * 100) / 100d == -.22);


When this test is run, I'm getting the following error

RemoteTransportException[[node_sm1][local[4]][indices:data/read/search[phase/dfs]]]; nested: ResourceNotFoundException[unable to find script [Score] using lang [native] in cluster state]; }
at __randomizedtesting.SeedInfo.seed([92F76D668DFB2FE9:BE6CD19FE52E4E36]:0)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:272)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:130)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:241)
at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:87)
at org.elasticsearch.action.search.InitialSearchPhase.access$100(InitialSearchPhase.java:47)
at org.elasticsearch.action.search.InitialSearchPhase$1.onFailure(InitialSearchPhase.java:155)
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:51)
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1050)
at org.elasticsearch.transport.local.LocalTransport.handleException(LocalTransport.java:467)
at org.elasticsearch.transport.local.LocalTransport.handleResponseError(LocalTransport.java:458)
at org.elasticsearch.transport.local.LocalTransport.processReceivedMessage(LocalTransport.java:338)
at org.elasticsearch.transport.local.LocalTransport.lambda$receiveMessage$0(LocalTransport.java:300)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: ResourceNotFoundException[unable to find script [Score] using lang [native] in cluster state]
at org.elasticsearch.script.ScriptService.getScriptFromClusterState(ScriptService.java:369)
at org.elasticsearch.script.ScriptService.compile(ScriptService.java:244)
at org.elasticsearch.script.ScriptService.search(ScriptService.java:490)
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:684)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:472)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:448)
at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:221)
at org.elasticsearch.action.search.SearchTransportService$5.messageReceived(SearchTransportService.java:328)
at org.elasticsearch.action.search.SearchTransportService$5.messageReceived(SearchTransportService.java:325)
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
at org.elasticsearch.transport.local.LocalTransport$2.doRun(LocalTransport.java:390)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
... 3 more

Which is caused by this line (used in the 1.7 tests)

.addScriptField("scores", new Script(ScriptType.STORED, "native", "Score", params))

I've tried using ScriptType.FILE and ScriptType.INLINE as well, but neither of them worked. The test is unable to load my script, and therefore unable to properly run the tests. Is there another method I should be using to access these scripts for the tests? Please let me know if you need any more information re: my set up. Any and all suggestions are welcome :grin:

IIRC it should be INLINE for native scripts. The error you get is because stored scripts are in the cluster state, which is not what you have. However, note that native scripts are going away in 6.0. Instead, you can see our docs on creating a script engine for advanced scripts.

Thanks for the reply, Ryan! Since writing this post, I did some further research and discovered the script engine for advanced scripts page you provided earlier. I opened up this topic because I'm having problems with accessing stored values from documents. My current workaround has been decoding the binary of the stored doc values, converting that to one very long string, and then using regex expressions to get the values I'm looking for, and that has made this script writing process very painful. I've read through the source code to see if there's another work around, but haven't had much luck. Could you possibly point me in the right direction there?

Sorry for the delayed reply, I have been out for a while. I recommend a couple things:

  • Use a real integration test (start an ES instance), don't use ESIntegTestCase or construct nodes in memory yourself. The only way to ensure your plugin works in a real ES cluster is with a real ES cluster.
  • Don't try to decode _source in a script. This is going to be orders of magnitude slower than you would want a scoring script to be. Instead, set the fields you want accessible as having doc values (most field types do by default).

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.