ESIntegTestCase 5.0 issues


#1

Hi,

I'm using gradle to test a small program to index and search documents. The program works fine but I wanted to add some unit tests using ESIntegTestCase (Before 5.0 I was creating a Node with the NodeBuilder which worked fine but I think that is no longer supported).

My test compile dependencies are as follows:

testCompile (group: 'org.apache.lucene', name: 'lucene-test-framework', version: '6.2.0')
testCompile (group: 'org.elasticsearch.test', name: 'framework', version: '5.0.0')
testCompile (group: 'org.apache.logging.log4j', name: 'log4j-core', version: '2.6.2')

I originally started off without log4j but had to add it because it complained of class not found. After adding log4j it then failed with a security manager error and had to add -Dtests.security.manager=false. The next error is:

org.elasticsearch.common.inject.ConfigurationException: Guice configuration errors:

  1. No implementation for org.elasticsearch.http.HttpServerTransport was bound.
    while locating org.elasticsearch.http.HttpServerTransport

The test class is as follows:

public class MyTest extends ESIntegTestCase {
@Test
public void doesNotWork() throws IOException {

    InetSocketAddress[] inetSocketAddresses = cluster().httpAddresses();
    for (InetSocketAddress s : inetSocketAddresses) {
        System.out.println(">" + s);
}

Any help would be much appreciated.


(David Pilato) #2

Here are the deps I'm basically adding in maven:

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.test</groupId>
            <artifactId>framework</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>transport-netty4-client</artifactId>
            <version>5.0.0</version>
            <scope>test</scope>
        </dependency>
        <!-- This optional dependency is used by the test framework -->
        <dependency>
            <groupId>net.java.dev.jna</groupId>
            <artifactId>jna</artifactId>
            <version>4.1.0</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>2.7</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.7</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>2.7</version>
            <scope>test</scope>
        </dependency>

Then some java test class:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE)
public abstract class MyIT extends ESIntegTestCase {
    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return Arrays.asList(Netty4Plugin.class);
    }

   // Tests here
}

I hope this helps.


#3

Thanks David, your answer was much appreciated. I thought overriding of the nodePlugins() method to include the Netty4Plugin would have solved my issue but it is still complaining. This only happens if I have a test that contains the following lines:

    InetSocketAddress[] inetSocketAddresses = cluster().httpAddresses();
    for (InetSocketAddress s : inetSocketAddresses) {
        System.out.println(">" + s);
    }

The reason I need to get hold of the httpAddresses is because I want to test some REST calls (not the binary API).

Cheers,

Stuart


(David Pilato) #4

Rest layer is disabled by default in IT. That's probably the reason here.

I can share later what I did for that but TBH it's better to run a real elasticsearch cluster to test the REST layer. See http://david.pilato.fr/blog/2016/10/18/elasticsearch-real-integration-tests-updated-for-ga/

HTH


#5

Thanks David, the link was very helpful.


(Jason Tedor) #6

The Log4j dependencies for Elasticsearch 5.0.0 should be 2.6.2, not 2.7 otherwise you will see NoClassDefFoundError for org.apache.logging.log4j.core.async.DaemonThreadFactory and possibly other issues.


(Eric Collins) #7

Hi David,

I get the idea that running a full fledged external cluster is more realistic, and I'd like to set that up for our continuous build stream where we have maven running tests in a segregated environment. But, I'm interested in being able to run unit tests from an IDE that use the JestClient without having to start up an external ES cluster. I see in your article the tests are effectively skipped if run from the IDE...

ESIntegTestCase seems so tantalizingly close to being able to do this, and the Jest guys even have this test class which is nearly exactly what I want to do:

They seem to be running an older version of ES though, 2.4 maybe. Back when nodes weren't set to local by default maybe? When I try to do something similar I get the below error when calling cluster().httpAddresses(). Obviously because the node is set to local... doing something simple like setting "NetworkModule.HTTP_ENABLED.getKey(), true" in the NodeSettings just gives a different error about binding types, so one would guess there's more to setting up the network stuff... Have you been able to get that to work within the context of ESIntegTestCase?

org.elasticsearch.common.inject.ConfigurationException: Guice configuration errors:

  1. No implementation for org.elasticsearch.http.HttpServerTransport was bound.
    while locating org.elasticsearch.http.HttpServerTransport

1 error
at __randomizedtesting.SeedInfo.seed([957850041CAE7925:3A6E197C03FB967E]:0)
at org.elasticsearch.common.inject.InjectorImpl.getProvider(InjectorImpl.java:790)
at org.elasticsearch.common.inject.InjectorImpl.getProvider(InjectorImpl.java:729)
at org.elasticsearch.common.inject.InjectorImpl.getInstance(InjectorImpl.java:801)
at org.elasticsearch.test.InternalTestCluster.getInstanceFromNode(InternalTestCluster.java:1205)
at org.elasticsearch.test.InternalTestCluster.getInstances(InternalTestCluster.java:1151)
at org.elasticsearch.test.InternalTestCluster.httpAddresses(InternalTestCluster.java:1216)

Thanks much,
Eric


(David Pilato) #8

What I do in that case:

  • Install elasticsearch x.y.z
  • Launch it with:
# ES >=  5.0
bin/elasticsearch -Ehttp.port=9400
# ES < 5.0
bin/elasticsearch -Des.http.port=9400
  • Run tests from the IDE

If you want to start the node from your tests you can indeed try to do something like:

@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE, numDataNodes = 1, numClientNodes = 0, transportClientRatio = 0, supportsDedicatedMasters = false)
public class YourTest extends ESIntegTestCase {

    @Override
    protected Settings nodeSettings(int nodeOrdinal) {
        int randomPort = randomIntBetween(49152, 65525);
        
        Settings.Builder builder = Settings.builder()
                .put(super.nodeSettings(nodeOrdinal))
                .put(NetworkModule.HTTP_ENABLED.getKey(), true)
                .put(HttpTransportSettings.SETTING_HTTP_PORT.getKey(), randomPort)
                .put("network.host", "127.0.0.1")
        Settings settings = builder.build();
        return settings;
    }

    @Override
    protected Collection<Class<? extends Plugin>> nodePlugins() {
        return Arrays.asList(Netty4Plugin.class);
    }
}

I think you may need to add this as well:

        <dependency>
            <groupId>org.elasticsearch.plugin</groupId>
            <artifactId>transport-netty4-client</artifactId>
            <version>${elasticsearch.version}</version>
            <scope>test</scope>
        </dependency>

HTH


(Eric Collins) #9

Hi David,

Sorry for the lengthy delay, but wanted to thank you for the response. I ended up switching to use the native client rather than JEST. The API ends up being nearly the same between native and JEST, but just one less part to worry about... and works much better with ESIntegTestCase...

Eric


(Pedro Calais Guerra) #10

Netty4Plugin works for me to connect to ESIntegTestCase nodes through HTTP,
the issue I have is that this operation seem to leave some live threads:

Mar 25, 2017 3:59:54 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
WARNING: Will linger awaiting termination of 1 leaked thread(s).

I suspect that ESIntegTestCase does not close all threads it opens for establishing HTTP connections with nodes, does it make sense?


(system) #11