Need to start ES instance for test purpose

(Prakhar Jain) #1

Hi Ivan and others,
I would like to discuss what I am doing, and need help regarding it.

We have two components - App (Jersey client) and Data (Elasticsearch)
We used to have Couchbase as a data component but we are migrating to elasticsearch.

In the App component, we serve data from elasticsearch.
I need to write tests on the App layer so that we conform to some standards.

I was trying to use Elasticsearch test framework (ESIntegTestCase). But it had lot of dependency issues.
Basically jar hell was creating problem and logger also.

As you guys are not supporting embedded elasticsearch instance.

And, the documentation is pointing me to use ESIntegTestCase
https://www.elastic.co/guide/en/elasticsearch/reference/current/integration-tests.html

I was in trouble since I wasn't able to integrate it my test due to issues mentioned above.

In this issue, nik9000 has mentioned that it is better to spin new instance rather than using ESIntegTestCase,

Alecojacoru elasticsearch maven plugin servers this purpose. But I wanted to use it in gradle project. I wasnt able to find any alternative for it.

Please help me find solution.
Thanks,
Prakhar

(David Pilato) #2

And you have also this (WIP):

(Prakhar Jain) #3

I am not able to build the above module.
Is it present in mvn central. Not able to get it from gradle.

Edit:
I tried mvn clean install, its failing on tests saying could not find valid docker environment.

I need to run this on simple machine. i dont have docker containers to run it on.

(Prakhar Jain) #4

I am trying to figure out how to run it on simple machine.
I have a gradle project, and by far the best way I have found it is using esplugin.

There I want to use integTestCluster as pre requisite for all my test. (i.e. I want to start elastic search instances before running tests on it)

What is the best way to do it?

(Prakhar Jain) #5

Hi, anyone from elasticsearch community, please reply!

One more question, we were earlier using embedded elastic search instance for JUnit tests.
What approach should we use now?

(Prakhar Jain) #6

Someone please!

BUMP

(David Pilato) #7

Read this and specifically the "Also be patient" part.

(Prakhar Jain) #8

sorry, but I need some solution urgently.
Don't meant to hurt anyone.

Thanks,
Prakhar

(David Pilato) #9

In the link I just pasted you can read:

There are no SLAs on responses to questions posted on this forum, if you require help with an SLA on responses you should look into purchasing a subscription package that includes support with an SLA such as those offered by Elastic.

(Ivan Brusic) #10

At this point, I seriously doubt anyone will answer your question. Elastic
support staff do not know anything about internals. Your only hope is for a
developer to chime in, but they are rarely here. The code is open source,
your best option is to dig in. Given the dependency issues, starting your
own instance is the only way to go.

(David Pilato) #11

@Ivan This is not how support is working. When someone in the support team does not know an answer, they are asking for help from the dev team. And there are SLAs.

Here we are all doing our best to answer. Some answers are easy. Some other can take more time.

And regarding the problem, I can't answer myself:

Because of my lack of knowledge so far on Gradle, I pinged in another thread @nik9000 and @rjernst who might have time to answer but again there is no SLA on this forum and it could take time for people to answer.

(Ryan Ernst) #12

As has been pointed out, the gradle elasticsearch.esplugin plugin is meant to develop a plugin. However, you can create an integTest task in your project which will launch an elasticsearch instance, and run REST tests. Try applying the elasticsearch.standalone-rest-test and elasticsearch.rest-test plugins. See projects under the qa dir in elasticsearch for examples.

(Ivan Brusic) #13

With the elasticsearch plugins, wouldn't the dependencies get pulled in? At
least in the test configuration. Was going to test it out
myself. ESIntegTestCase still works for me whenever I do not have
dependency conflicts (always Lucene).

Let's face it, if it weren't for Ryan, none of these types of questions, at
least mine, will ever be answered. :slight_smile: Thanks Ryan!

(Prakhar Jain) #14

You guys are using ESTestCase, ESIntegTestCase in it.
This means, I will get those jar hell issues.

(Ryan Ernst) #15

@Ivan By "elasticsearch plugins" do you mean applying elasticsearch.esplugin? If so, then yes, you get an integTest automatically by using that. But you can also create a rest test (ie tasks for starting an ES cluster and test task to run the REST test runner) directly in a project. This is how many projects under the qa directory in elasticsearch work.

@prakharjain17 I am not talking about ESTestCase or ESIntegTestCase. What I suggested would allow you to start a real elasticsearch node, and create REST tests against it (this are the high level tests we use to check the elasticsearch apis). You can also create a test in java to connect the cluster. Look at the client qa test. That particular test uses the transport client, but you could replace the example dependency there with the rest client if that is what you are using.

(Prakhar Jain) #16

Hi, I am having this gradle file

buildscript {
    repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
    }

    dependencies {
        classpath "org.elasticsearch.gradle:build-tools:6.1.1"

    }
}

apply plugin: 'idea'
apply plugin: 'java'
apply plugin: 'elasticsearch.standalone-rest-test'
apply plugin: 'elasticsearch.rest-test'


dependencies {
// https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client
    compile group: 'org.elasticsearch.client', name: 'elasticsearch-rest-high-level-client', version: '6.1.1'

}

with following error

Cannot add a SourceSet with name 'test' as a SourceSet with that name already exists.
(Prakhar Jain) #17

its getting compiled for this gradle file, but I need to apply the plugin java. Once I have built the gradle file. How to use elasticsearch instance from my code i.e. how to get client from it or else hostname and port.

buildscript {
    repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
    }

    dependencies {
        classpath "org.elasticsearch.gradle:build-tools:6.1.1"

    }
}
apply plugin: 'eclipse'
apply plugin: 'idea'
apply plugin: 'elasticsearch.standalone-rest-test'
apply plugin: 'elasticsearch.rest-test'

repositories {
    mavenLocal()
    mavenCentral()
    jcenter()
}
sourceSets {
    main {
        java {
            srcDirs = ['src/java']
        }
        resources {
            srcDirs = ['src/resources']
        }
    }
    test {

    }
}

dependencies {
// https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-high-level-client
    compile group: 'org.elasticsearch.client', name: 'elasticsearch-rest-high-level-client', version: '6.1.1'

}
(David Pilato) #18

In case it helps, you can read this post from @joel-costigliola :

Which leads to

(Prakhar Jain) #19

Hi dadoodnet,
I have already built this kind of sample project myself.
The problem is integrating the test framework in your own project.
It creates jar hell problem and logger problem.

Anyways, thanks for the reply.

(system) closed #20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.