Starting up InternalTestCluster outside of ESIntegTestCase

Elastic Stack Elasticsearch
1

Hi, I want to write an integration test against and embedded ES test cluster.

I read about ESIntegTestCase and managed to successfully implement a FooTest extends ESIntegTestCase. Cluster started (as expected), and I was able to index and search. All is good.

However, I have an issue with needing to extend ESIntegTestCase as I also need to extend my own BaseTestCase class. To give some more context, I have an interface (let's call it Foo) with several implementations, one of which is ElasticClientFoo which implements the Foo API by communicating with an ES cluster via Client. I intend to write a FooTestBase with the test methods, with actual extensions providing the specific Foo implementation. So for example, I would have FooElasticClientTest extends FooTestBase which will override say one method which initializes the ElasticClientFoo with the client() provided by InternalTestCluster.

I tried implementing a MiniElasticTestCluster which initializes an InternalTestCluster, and I even managed to get it working. However when I added indexing and search to my tests, they started to fail on ThreadLeakage errors. Since I'm familiar with Lucene's randomized test framework, I can partially explain it -- since I do not extend ESIntegTestCase, my test does not hook properly with RandomizedRunner, causing its @After to get called before my @AfterClass (which shuts down the cluster), and that causes RandomizedRunner to report the threads leakage, even though the cluster is shut down properly afterwards.

Is there a way to initialize InternalTestCluster, or any other instance, for test purposes, even though I don't care about randomizing the tests? I am looking for something similar to Solr's MiniSolrCloudCluster or ZooKeeper's (actually curator's) TestingServer, which will allow me to fire up an ES cluster, embedded for testing purposes?

If not, how difficult do you think it would be to adapt InternalTestCluster to do that, without tying to the randomization framework?

P.S. I do not rule out a problem with my implementation which wraps InternalTestCluster that causes these thread leaks, so if anyone has already done something like this outside of ES, I'd love to get some pointers / suggestions.

2

Rather than extends ESIntegTestCase you are likely better off using an external Elasticsearch cluster for your testing. ESIntegTestCase is great for really really invasive testing of the Elasticsearch but usually you don't need the level of control that it offers and you'd be better of with a real Elasticsearch node without the dependency on whatever hacks we add to get that control.

Here is a link for a maven plugin that starts Elasticsearch as part of a build. This also exists but it doesn't support 5.0 yet.

3

Thanks for the quick response @nik9000! I did read about that plug-in a couple days ago but decided to try the embedded path for two main reasons, both are related to debugging. Solr's MiniSolrCloudCluster allows you to debug your test including stepping into Solr's code and understanding why a test fails. This is most valuable as sometimes (well most of the times) it reveals a bug in my test/code, but some times it also reveals a bug in Solr.

Another reason is that it's just easier to run a test from an IDE (Eclipse in my case), add debug print statements, stepping through own code, than running a test in Maven.

I still hope to manage getting the embedded case to work, mostly for the above reasons (but also since now it has become a challenge that I must solve! :slight_smile:). If it won't work though, I'll explore the Maven plug-in option.

4

@nik9000 just wanted to bring closure to this topic. The Ant script worked well and allowed me to start an ES cluster before my integration tests start.

I had some issues running the script on Windows though, as it passes the arguments to the cluster w/ -D (JVM system properties), but on Windows I had to specify them using -E. This seems to work both on Windows and Linux.

I don't mind submitting those changes as a PR. Can you point me to instructions on how to do that (if you're interested of course).

5

I think @david is required to change those scripts, at least that is my
guess off hand.

6

Hey @nik9000 (or @david) - I wanted to create a PR with my changes to integration-tests.xml but I can't find that file in 5.x. Was it removed?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.