Integration testing in Java, carrotsearch thread leaks & severe errors

Elastic Stack Elasticsearch
1

Every time I run my integration test (which extends ElasticsearchIntegrationTest) I get a thread leak severe log message from carrotsearch. What I've found in googling for a resolution to this suggests that this was fixed a long time ago. But I'm still seeing it with version 1.7.0 of org.elasticsearch:elasticsearch (regular dependency as well as test-jar), 2.1.13 of com.carrotsearch.randomizedtesting:randomizedtesting-runner and 4.10.4 of org.apache.lucene:lucene-test-framework. I've also tried with 1.7.1, 2.1.16 and 5.2.1 respectively but run into other dependency issues when I do that.

My test base class is set up like so:

@ElasticsearchIntegrationTest.ClusterScope(scope= ElasticsearchIntegrationTest.Scope.TEST)
abstract public class ReportingFullStackIT extends ElasticsearchIntegrationTest {
...
}

And the exception I get is this:

Aug 04, 2015 12:13:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
WARNING: Will linger awaiting termination of 5 leaked thread(s).
Aug 04, 2015 12:13:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
SEVERE: 2 threads leaked from SUITE scope at com.foo.stuff.MyTestIT:

... abbreviated

Aug 04, 2015 12:13:20 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
INFO: Starting to interrupt leaked threads:
   1) Thread[id=88, name=metrics-meter-tick-thread-1, state=TIMED_WAITING, group=TGRP-MyTestIT]
   2) Thread[id=89, name=metrics-meter-tick-thread-2, state=WAITING, group=TGRP-MyTestIT]
Aug 04, 2015 12:13:23 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
SEVERE: There are still zombie threads that couldn't be terminated:

... abbreviated

Throwable:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:

... abbreviated

        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([C0B4BC6EAE06585B]:0)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    __randomizedtesting.SeedInfo.seed([C0B4BC6EAE06585B]:0)

... abbreviated

   2) Thread[id=89, name=metrics-meter-tick-thread-2, state=WAITING, group=TGRP-MyTestIT]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
    __randomizedtesting.SeedInfo.seed([C0B4BC6EAE06585B]:0)

Process finished with exit code 255

From what I can tell, it looks like the issue is in __randomizedtesting.SeedInfo.seed. I don't understand why this is happening though.

Is it possible that the docs are wrong?
https://www.elastic.co/guide/en/elasticsearch/reference/1.6/using-elasticsearch-test-classes.html

2

I'm also having this exact issue. Have you been able to solve, or work around, it?

Cheers
/Marcus

3

I'm also getting that in two cases:

  • my test fails and somehow some threads are not stopped. Basically fix the test and this error will disappear
  • I added some static variables within the test class using @BeforeClass. At the end of the test, you need to create an @AfterClass and set static variable to null. Also, if you started some processes within the test, make sure you are stopping all of them.

If possible, avoid any static initialization in your test class.

Hope this helps

4

I had the same issue, and was able to fix it by adding the following to the test class...

@ThreadLeakScope(ThreadLeakScope.Scope.NONE)

I got the answer from this github issue, https://github.com/elastic/elasticsearch/issues/8642