Hi all,
I am seeing an issue running consecutive tests that startup 2 elastic
search clusters in the setup for the Test Class and close the clients and
nodes down when the Test Class finishes. Individually by themselves the
tests run clean. It seems that during the closure of the client and node,
there seems to be some state or something else that persists, which affects
the tests in the next consecutive Test Class which does the same thing in
the setup and teardown for the class. Note that I am not doing this for
each test method. I am using TestNG, so I am writing the setup and teardown
myself and not using the Junit-based ElasticSearchTest classes that come
with the standard distribution. They are supposed to be 2 single-node
clusters.
Here are the 2 kinds of failures I am seeing, which leads to a few test
failures:
java.lang.RuntimeException: java.util.concurrent.TimeoutException * (Basically
http requests to the cluster instance are timing out)*
at
org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:226)
org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m]
at
org.elasticsearch.action.support.master.TransportMasterNodeOperationAction$4.onTimeout(TransportMasterNodeOperationAction.java:164)
at
org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:239)
at
org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:520)
Here is how my setup and teardown looks like:
Setup calls setupElasticSearch()
Teardown calls closeDownElasticSearch()
Type is an enum. The methods below are utility methods that are called in
each test class. Any insight will be appreciated.
public static String setupElasticSearch()
throws Exception
{
if (setupES.compareAndSet(false, true)) {
randomSuffix = getRandomString();
Reporter.log("Setting up Elastic Search", true);
createESCluster(randomSuffix, type1);
createESCluster(randomSuffix, type2);
}
return randomSuffix;
}
private static void createESCluster(String randomSuffix, Type type)
throws Exception
{
Settings settings = getSettings(randomSuffix, type);
setupNodeAndClient(randomSuffix, type, settings);
}
private static void ensureGreen(Client client)
throws Exception
{
ClusterHealthResponse actionGet = client.admin().cluster()
.health(Requests.clusterHealthRequest("ids*").timeout(new
TimeValue(30,
TimeUnit.SECONDS)).waitForGreenStatus().waitForEvents(Priority.LANGUID).waitForRelocatingShards(0)).actionGet();
if (actionGet.isTimedOut()) {
logger.info("ensureGreen timed out, cluster state:\n{}\n{}",
client.admin().cluster().prepareState().get().getState().prettyPrint(),
client.admin().cluster().preparePendingClusterTasks().get().prettyPrint());
throw new Exception("Could not set cluster to green");
}
}
private static void setupNodeAndClient(String randomSuffix, Type type,
Settings settings)
throws Exception
{
if (type == type1) {
type1Node = buildNode(randomSuffix, settings, type);
type1Client = type1Node.client();
initNode(type1Client);
ensureGreen(type1Client);
}
else if (type == type2) {
type2Node = buildNode(randomSuffix, settings, type);
type2Client = type2Node.client();
initNode(type2Client);
ensureGreen(type2Client);
}
}
public static void initNode(Client client)
throws Exception
{
PutIndexTemplateResponse response = client.admin().indices()
.preparePutTemplate("es_1")
.setTemplate("es*")
.setSettings(Resources.toString(Resources.getResource("es-schema-template-settings.json"),
Charsets.UTF_8))
.addMapping("message",
Resources.toString(Resources.getResource("es-schema-template-mappings.json"),
Charsets.UTF_8))
.execute()
.get();
if (!response.isAcknowledged()) {
throw new Exception("Template PUT did not succeed.");
}
}
private static Node buildNode(String randomSuffix, Settings settings,
Type type)
{
return NodeBuilder.nodeBuilder()
.clusterName(ES_TEST + "-" + type.name().toLowerCase() +
"_" + randomSuffix)
.settings(settings)
.data(true)
.node();
}
private static Settings getSettings(String randomSuffix, Type type)
{
String suffix = randomSuffix + "-" + type.name().toLowerCase() +
"/";
return ImmutableSettings.settingsBuilder()
.put("path.data", "data" + suffix)
.put("path.conf", "conf" + suffix)
.put("path.work", "work" + suffix)
.put("discovery.initial_state_timeout", "3s")
.put("gateway.type", "none")
.put("index.store.type", "memory")
.put("index.store.fs.memory.enabled", true)
.build();
}
public static void closeDownElasticSearch(String randomSuffix)
throws Exception
{
if (setupES.get()) {
if (highClient != null || normalClient != null) {
closeClientAndNode(randomSuffix, type1, type1Node,
type1Client);
closeClientAndNode(randomSuffix, type2, type2Node,
type2Client);
if (FileUtils.deleteRecursively(new
File("data-indexable-data-store-test-high"))) {
Reporter.log("Removed data/");
}
if (FileUtils.deleteRecursively(new
File("data-indexable-data-store-test-normal"))) {
Reporter.log("Removed data/");
}
}
}
}
private static void closeClientAndNode(String randomSuffix, Type type,
Node node, Client client)
throws Exception
{
DeleteIndexResponse delete = client.admin().indices().delete(new
DeleteIndexRequest("es*")).actionGet();
if (!delete.isAcknowledged()) {
logger.error("Index wasn't deleted");
throw new Exception("Index wasn't deleted");
}
client.close();
if (!node.isClosed()) {
node.close();
}
String suffix = randomSuffix + "-" + type.name().toLowerCase() +
"/";
FileUtils.deleteRecursively(new File("data" + suffix));
FileUtils.deleteRecursively(new File("conf" + suffix));
FileUtils.deleteRecursively(new File("work" + suffix));
}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/67451697-428f-4d72-952d-918ed4a5d465%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.