Elasticsearch.esplugin - ES Gradle Build Tools: integTestRunner failing

[Thales_Valias]

Context:

I'm trying to implement an ES Ingest Processor plugin through this cookiecutter template: GitHub - spinscale/cookiecutter-elasticsearch-ingest-processor: A cookiecutter template for an elasticsearch ingest processor plugin
following this post: Writing Your Own Ingest Processor for Elasticsearch | Elastic Blog

Environment:

MacOS Mojave 10.14.2
openjdk 11
Gradle 5.2.1 (wrapped)
Intellij 2018.3.4 (or directly on Terminal)
org.elasticsearch.gradle:build-tools:6.6.0
Third Party library - org.nibor.autolink:autolink:0.6.0 (just like the example on the blog post above)

Problem

I can't get my project to work through all the

./gradlew check

tasks properly when calling a method of a third party library (which is set as a dependency), ending up with a failure in integTestRunner with the following exceptions:

Throwable #1: java.lang.RuntimeException: Failure at [ingest-url-extract/20_url-extract_processor:3]: Connection closed

Caused by: java.net.ConnectException: Connection refused

[2019-02-18T19:37:18,932][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-0] fatal error in thread [elasticsearch[node-0][management][T#1]], exiting
java.lang.NoClassDefFoundError: org/nibor/autolink/LinkType

I got those same errors when I try to call methods of different dependencies as well. The unitTest task works well, though (even calling the same method).

If I change

implementation 'org.nibor.autolink:autolink:0.6.0' or
api 'org.nibor.autolink:autolink:0.6.0'

to

compile 'org.nibor.autolink:autolink:0.6.0'

it works. However, Gradle discourage the use of it, since it's now deprecated:

The compile, testCompile, runtime and testRuntime configurations inherited from the Java plugin are still available but are deprecated. You should avoid using them, as they are only kept for backwards compatibility.

Also on my real project which I use an internal dependency (a big one) compile doesn't do the trick. I got a NoClassDefFoundError earlier. autolink doesn't have any transitive dependency.

Edit: with 'compile' the transitive dependencies aren't taken into the project. With 'implementation' they are. I think that's why I get a NoClassDefFoundError earlier when using the huge internal dependency on my project.

Edit 2: I tested other project and libraries, like this one:
GitHub - spinscale/elasticsearch-ingest-langdetect: Ingest processor doing language detection for fields

And noticed that if I use 'compile' on its dependencies, it works correctly, even not bringing its transients. But if I change it to 'implementation' or 'api', it doesn't work. I can't get why.

Project Configuration

This is my build.gradle file:

    buildscript {
      repositories {
        mavenLocal()
        mavenCentral()
        jcenter()
      }

      dependencies {
        classpath "org.elasticsearch.gradle:build-tools:6.6.0"
      }
    }

group = 'org.elasticsearch.plugin.ingest'
version = '0.0.1-SNAPSHOT'

apply plugin: 'java-library'
apply plugin: 'idea'
apply plugin: 'elasticsearch.esplugin'

licenseFile = rootProject.file('LICENSE.txt')
noticeFile = rootProject.file('NOTICE.txt')

esplugin {
  name 'ingest-url-extract'
  description 'Ingest processor that is doing something awesome'
  classname 'org.elasticsearch.plugin.ingest.url.extract.IngestUrlExtractPlugin'
  licenseFile rootProject.file('LICENSE.txt')
  noticeFile rootProject.file('NOTICE.txt')
}

repositories {
  mavenLocal()
  mavenCentral()
  jcenter()
}

dependencies {
  // Also tried 'api' instead, without success
  implementation 'org.nibor.autolink:autolink:0.6.0'
}

checkstyleMain.enabled = false
checkstyleTest.enabled = false
dependencyLicenses.enabled = false
thirdPartyAudit.enabled = false

Other Information

My project is pretty much identical to this one: Writing Your Own Ingest Processor for Elasticsearch | Elastic Blog

The problem occurs during integTestRunner, which attempts to test against ES real rest service. Everything prior it works well.

What I'd like

To understand why this simple project can't execute all the ./gradlew check tasks successfully when I use third parties methods from dependencies defined with implementation or api configuration.

To understand why compile is the only config that works and only when a dependency doesn't rely on transitive dependencies.

I'd appreciate a lot if anyone could give me a hand. I've been stuck on this for a while. Thanks!!

[Thales_Valias]

I decided to go with maven instead of gradle since I'd have more control over things (I'm using a huge dependency and I wasn't being able to track the issues from es gradle build tools, unfortunately).

Now I'm having more success and could move forward. I suspect I was having problems related to permissions (java-security), although I was running with it disabled.

[loren (Loren Siebert)]

Not sure if this is helpful, but I have an ingest plugin based on Cookiecutter and Gradle 5.2 here.

IIRC, for integration tests to pass I had to modify the Java security policy like this.

Good luck!

[Thales_Valias]

First of all, thanks Loren for your contribution.

I didn't have any problem with maven. I tried to go back to Gradle with the build tools and mimic what worked with maven, including all the permissions and jar management, but the problem still persists.

As I've said before, the dependency which my project relies on is quite big, composed by more than 20 modules. I believe there's some hidden problem happening with that that build tools complains, but maven doesn't. Although I don't have any meaningful msg error out of it (just a java.lang.NoClassDefFoundError).

Anyway, as maven is doing the trick, no worries. I just wanted to leave here the way I took.