Embedded integration tests in 5.x - missing plugin mvn artifacts

Elastic Stack Elasticsearch
1

In reading https://www.elastic.co/blog/elasticsearch-the-server - I understand that embedded ES isn't a supported configuration. While it's easy enough to add this functionality back in myself, there doesn't seem to be a way to pull in ES plugins.

My use case is to: start up an embedded ES for integration tests, configure it with checked in settings and mappings and index a test set of documents. This way I can fully test the various queries my system generates against known data, which is really helpful since regressions here aren't acceptable to my users.

I know I could treat ES like an external DB and run tests against that, but it'd be unfortunate to have to introduce all the environment-related issues which come along with this approach.

Is it possible to start publishing plugin artifacts to a maven repository again?

2

Just use maven elasticsearch plugin. It helps very much to solve this Use case.

3

Thanks, this one? https://github.com/alexcojocaru/elasticsearch-maven-plugin

4

Exactly

5

Missed the whole 'maven' part - unfortunately we're on gradle.

Seems odd to push this into external buildtools (this one is 3k sloc!). Is there any reason ES no longer publishes plugin artifacts?

Another usecase I've hit in my upgrade: I have a custom query plugin - ideally I'd like to test it against the analyzers I use in prod. But some analyzers (eg. phonetic) are pulled in via plugins.

6

So if you are on gradle you can probably reuse the gradle tasks we built?

May be this can help?

https://github.com/elastic/elasticsearch/blob/master/buildSrc/src/main/groovy/org/elasticsearch/gradle/test/RestIntegTestTask.groovy

When I'm building a plugin and I want to run integration tests, I'm using ATM a ant script (which is really similar to what maven plugin is doing - I'll switch to the later soon). The ANT script does:

  • Download elasticsearch
  • Optionally install x-pack
  • Install the plugin I'm building
  • Start elasticsearch

After the tests, it shutdowns the node.

Here are some blog posts about it: https://david.pilato.fr/blog/2016-08-03-elasticsearch-real-integration-tests-with-security-enabled/

You can still see all that in action at GitHub - dadoonet/fscrawler: Elasticsearch File System Crawler (FS Crawler)

Sorry if this is not super helpful but at least it can give some ideas.

Jars are not needed anymore as only zip files should be installed by the plugin manager.
Zip can be downloaded as artifacts from elasticsearch download service. Have a look at:

and

I hope this helps.

7

I've figured out how to do this in gradle. It's not pretty but it gives me a compile dependency on analysis-phonetic. I think this'll break down if I pull in, transitively or otherwise, dependencies which are provided by the plugin zip.

So it works, but it's a hack. Is there no chance of just going back to publishing these artifacts? I can't be the only one who needs a compile time dependency on plugins.

8

You're certainly not the only one. I reported the same thing in the past. It seems like the artifacts were taken down because Elastic think nobody should need them, without actually considering whether some of us were using them.

Even if Elastic want to maintain their own repository to host the files, you'd think they could at least make the structure the same as a real Maven repository so that we can continue to use normal methods of the build tool to download dependencies.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.