Need help grokking gradle build

Elastic Stack Elasticsearch
1

I am adding a test fixture to a plugin of mine and I am using the
elasticsearch build tools defined in the build plugin. I am using the
example-fixture as a guide:

The plugin is recognized correctly and things work well until the Forbidden
APIs check:

Task :test/fixtures/db-fixture:forbiddenApisMain
[snip]
build-tools/5.4.1/f18e5d8b1efc699ab21ec3863bf8fae24a5b93e8/build-tools-5.4.1.jar!/forbidden/es-all-signatures.txt

FAILURE: Build failed with an exception.

  • What went wrong:
    Execution failed for task ':test/fixtures/db-fixture:forbiddenApisMain'.

Parsing signatures failed: Class 'org.apache.lucene.index.IndexReader'
not found on classpath while parsing signature:
org.apache.lucene.index.IndexReader#getCombinedCoreAndDeletesKey()

Obviously the classpath is not defined correctly for the fixture. Since the
gradle build process in elasticsearch is top-down, the example-fixture is
meaningless outside of the entire elasticsearch gradle build context. I am
no gradle maven (pun intended), but this step should be easy enough to
figure out. :slight_smile: Trying to figure out where in build.gradle the dependencies
are being set for the dependencies. Perhaps it is inside one of the
plugins. Does anyone knowledgable of the build process have an idea? Basing
things off of 5.4 for now.

Cheers,

Ivan

2

We actually have forbidden apis, among other things, disabled for fixtures. You can see this in test/build.gradle.

3

Thanks Ryan. I was hoping you would chime in since you provided some good
input last time (finally have a chance to revisit this project).

Totally missed the fact that :test was a sub-project. I see the fixtures
getting included in settings.gradle, but not :test. Since I am learning
about the build, where is that done? In the end, I disabled many tasks:

dependencyLicenses.enabled = false
forbiddenApisMain.enabled = false
jarHell.enabled = false
licenseHeaders.enabled = false
test.enabled = false
thirdPartyAudit.enabled = false

Now comes the fun part of kicking off the fixture before running the tests!
I'll try to compile my notes on how to create a minimal plugin/fixture
outside of the elasticsearch build context. Of course, things might change
with 6.x.

Cheers,

Ivan

4

You're one of the first people I've heard of doing this. so we would greatly value your feedback on the process when you have it all working. I would love to have a more complete section in the docs for plugin authors, especially in regards to how to test.

5

I will gladly play the role of the guinea pig. At least this time there is
a team to answer questions, although things were easier when it was only
Shay. :slight_smile: Hopefully you are still on the line.

Trying to create a minimalist fixture. Come to find out that the pid and
ports files are required when using an AntFixture. The pid file I can
understand, but the ports file is not always necessary. I guess one task I
can document is what is expected by AntFixture. The plugins are dependent
on the fixtures to create the files, which is a weird dependency. What if
there are no fixtures, or more than one?

Even starting off with a near empty fixture causes failures. Either the
fixture stalls during Thread.sleep with no integration tests run, or the
integration cluster fails to start. Occasionally the code runs without any
modifications.

Setting system properties in integTestRunner has no effect, unlike the
elasticsearch plugins, which means the task is not run. The closure seems
to have run. Something is being set by the parent build, but I cannot find
it.

Upgraded to elasticsearch 5.5.2 for the jstack debugging output, but not to
5.6 since I do not want to deal with another check (license). Very strange
that the code will alternate between stalling on the task, failing to start
the cluster, and just working (the rare event). The fixture does not need
the elasticsearch build plugin since it should be simply a Java program.

https://gist.github.com/brusic/c312b5aa708e564d100a8b1fa60b3388

Any insights?

6

I'm playing quite a lot with plugins myself and there are some undocumented prerequisites.

It's been a while but iirc for integTests to run and pass you have to put some yml files declaring at least one REST API test (under resources/rest-api-spec/test/) e.g. https://github.com/BigDataBoutique/elasticsearch-repository-swift/tree/master/src/test/resources/rest-api-spec/test. And you have to have at least one IT and the IntegTest class, see here: https://github.com/BigDataBoutique/elasticsearch-repository-swift/tree/master/src/test/java/org/wikimedia/elasticsearch/swift

7

Thanks Itamar for responding to this somewhat old thread.

I was able to get almost everything running, the only thing that I have but
figured out is integration testing and jar hell. I wrote up my findings
here: https://github.com/brusic/plugin-with-fixture-skeleton

I came across the testing requirements, but did not need any REST test, but
that is perhaps I did not create new REST endpoints in this example.

With the plugin gradle plugin, there is a call to the jar task that I
cannot seem to override in order to create an uber jar.

Cheers,

Ivan

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.