ESIntegTestCase 5.0 issues

trisport88 · October 27, 2016, 5:01am

Hi,

I'm using gradle to test a small program to index and search documents. The program works fine but I wanted to add some unit tests using ESIntegTestCase (Before 5.0 I was creating a Node with the NodeBuilder which worked fine but I think that is no longer supported).

My test compile dependencies are as follows:

testCompile (group: 'org.apache.lucene', name: 'lucene-test-framework', version: '6.2.0')
testCompile (group: 'org.elasticsearch.test', name: 'framework', version: '5.0.0')
testCompile (group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.6.2')

I originally started off without log4j but had to add it because it complained of class not found. After adding log4j it then failed with a security manager error and had to add -Dtests.security.manager=false. The next error is:

org.elasticsearch.common.inject.ConfigurationException: Guice configuration errors:

No implementation for org.elasticsearch.http.HttpServerTransport was bound.
while locating org.elasticsearch.http.HttpServerTransport

The test class is as follows:

public class MyTest extends ESIntegTestCase {
@Test
public void doesNotWork() throws IOException {

    InetSocketAddress[] inetSocketAddresses = cluster().httpAddresses();

    for (InetSocketAddress s : inetSocketAddresses) {
        System.out.println(">" + s);
}

Any help would be much appreciated.

dadoonet · October 27, 2016, 9:04am

Here are the deps I'm basically adding in maven:

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.test</groupId>
            <artifactId>framework</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>transport-netty4-client</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <!-- This optional dependency is used by the test framework -->
        <dependency>
            <groupId>net.java.dev.jna</groupId>
            <artifactId>jna</artifactId>
            <version>4.1.0</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.7</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.7</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.7</version>
            <scope>test</scope>
        </dependency>

Then some java test class:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE)
public abstract class MyIT extends ESIntegTestCase {
    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return Arrays.asList(Netty4Plugin.class);
    }

   // Tests here
}

I hope this helps.

trisport88 · October 27, 2016, 4:00pm

Thanks David, your answer was much appreciated. I thought overriding of the nodePlugins() method to include the Netty4Plugin would have solved my issue but it is still complaining. This only happens if I have a test that contains the following lines:

    InetSocketAddress[] inetSocketAddresses = cluster().httpAddresses();
    for (InetSocketAddress s : inetSocketAddresses) {
        System.out.println(">" + s);
    }

The reason I need to get hold of the httpAddresses is because I want to test some REST calls (not the binary API).

Cheers,

Stuart

dadoonet · October 27, 2016, 7:56pm

Rest layer is disabled by default in IT. That's probably the reason here.

I can share later what I did for that but TBH it's better to run a real elasticsearch cluster to test the REST layer. See http://david.pilato.fr/blog/2016/10/18/elasticsearch-real-integration-tests-updated-for-ga/

HTH

trisport88 · October 27, 2016, 8:41pm

Thanks David, the link was very helpful.

jasontedor · November 4, 2016, 12:26pm

The Log4j dependencies for Elasticsearch 5.0.0 should be 2.6.2, not 2.7 otherwise you will see NoClassDefFoundError for org.apache.logging.log4j.core.async.DaemonThreadFactory and possibly other issues.

Eric_Collins · December 16, 2016, 4:14am

Hi David,

I get the idea that running a full fledged external cluster is more realistic, and I'd like to set that up for our continuous build stream where we have maven running tests in a segregated environment. But, I'm interested in being able to run unit tests from an IDE that use the JestClient without having to start up an external ES cluster. I see in your article the tests are effectively skipped if run from the IDE...

ESIntegTestCase seems so tantalizingly close to being able to do this, and the Jest guys even have this test class which is nearly exactly what I want to do:

github.com

searchbox-io/Jest/blob/master/jest/src/test/java/io/searchbox/client/JestClientFactoryIntegrationTest.java

package io.searchbox.client;

import io.searchbox.client.config.HttpClientConfig;
import io.searchbox.client.http.JestHttpClient;
import io.searchbox.cluster.Health;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.pool.PoolStats;
import org.elasticsearch.common.network.NetworkModule;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.transport.Netty4Plugin;
import org.junit.Test;

import java.io.IOException;
import java.lang.reflect.Field;
import java.util.Collection;
import java.util.Collections;
import java.util.concurrent.TimeUnit;

This file has been truncated. show original

They seem to be running an older version of ES though, 2.4 maybe. Back when nodes weren't set to local by default maybe? When I try to do something similar I get the below error when calling cluster().httpAddresses(). Obviously because the node is set to local... doing something simple like setting "NetworkModule.HTTP_ENABLED.getKey(), true" in the NodeSettings just gives a different error about binding types, so one would guess there's more to setting up the network stuff... Have you been able to get that to work within the context of ESIntegTestCase?

org.elasticsearch.common.inject.ConfigurationException: Guice configuration errors:

No implementation for org.elasticsearch.http.HttpServerTransport was bound.
while locating org.elasticsearch.http.HttpServerTransport

1 error
at __randomizedtesting.SeedInfo.seed([957850041CAE7925:3A6E197C03FB967E]:0)
at org.elasticsearch.common.inject.InjectorImpl.getProvider(InjectorImpl.java:790)
at org.elasticsearch.common.inject.InjectorImpl.getProvider(InjectorImpl.java:729)
at org.elasticsearch.common.inject.InjectorImpl.getInstance(InjectorImpl.java:801)
at org.elasticsearch.test.InternalTestCluster.getInstanceFromNode(InternalTestCluster.java:1205)
at org.elasticsearch.test.InternalTestCluster.getInstances(InternalTestCluster.java:1151)
at org.elasticsearch.test.InternalTestCluster.httpAddresses(InternalTestCluster.java:1216)

Thanks much,
Eric

dadoonet · December 17, 2016, 8:18am

What I do in that case:

Install elasticsearch x.y.z
Launch it with:

# ES >=  5.0
bin/elasticsearch -Ehttp.port=9400
# ES < 5.0
bin/elasticsearch -Des.http.port=9400

Run tests from the IDE

If you want to start the node from your tests you can indeed try to do something like:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE, numDataNodes = 1, numClientNodes = 0, transportClientRatio = 0, supportsDedicatedMasters = false)
public class YourTest extends ESIntegTestCase {

    @Override
    protected Settings nodeSettings(int nodeOrdinal) {
        int randomPort = randomIntBetween(49152, 65525);
        
        Settings.Builder builder = Settings.builder()
                .put(super.nodeSettings(nodeOrdinal))
                .put(NetworkModule.HTTP_ENABLED.getKey(), true)
                .put(HttpTransportSettings.SETTING_HTTP_PORT.getKey(), randomPort)
                .put("network.host", "127.0.0.1")
        Settings settings = builder.build();
        return settings;
    }

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return Arrays.asList(Netty4Plugin.class);
    }
}

I think you may need to add this as well:

        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>transport-netty4-client</artifactId>
            <version>${elasticsearch.version}</version>
            <scope>test</scope>
        </dependency>

HTH

Eric_Collins · January 9, 2017, 6:53pm

Hi David,

Sorry for the lengthy delay, but wanted to thank you for the response. I ended up switching to use the native client rather than JEST. The API ends up being nearly the same between native and JEST, but just one less part to worry about... and works much better with ESIntegTestCase...

Eric

Pedro_Calais_Guerra · March 25, 2017, 7:01pm

Netty4Plugin works for me to connect to ESIntegTestCase nodes through HTTP,
the issue I have is that this operation seem to leave some live threads:

Mar 25, 2017 3:59:54 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
WARNING: Will linger awaiting termination of 1 leaked thread(s).

I suspect that ESIntegTestCase does not close all threads it opens for establishing HTTP connections with nodes, does it make sense?