Elasticsearch plugins missing from maven central

[Graham_Leggett]

Elasticsearch v6.2.2 is available from maven central:

https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch/6.2.2/

Elasticsearch plugins, like elasticsearch-analysis-icu plugin, are not available from maven central:

https://repo1.maven.org/maven2/org/elasticsearch/elasticsearch-analysis-icu/ (up to v2.7.0)
https://repo1.maven.org/maven2/org/elasticsearch/plugin/analysis-icu/ (up to 5.0.0-alpha5)

This makes it difficult to inject elasticsearch artifacts into maven projects.

I raised this as a bug at https://github.com/elastic/elasticsearch/issues/28765, however this bug was closed pending discussion, which has confused me. Is there an intention to keep publishing artifacts to maven central, or is the intention to stop doing so? If the elasticsearch project doesn't plan on publishing artifacts to maven central moving forward, what repository can I source the elasticsearch artifacts and plugins from?

[dadoonet (David Pilato)]

You don't need to add this plugin to your maven project. Those plugins are only meant to run inside elasticsearch servers.

What is your use case?

[Graham_Leggett]

Elasticsearch publishes debian and rpm packages for the main elasticsearch server, however none of the plugins have been packaged. We operate in an environment which offers a service level agreement, and so a best effort install process does not work for us.

We tried to contribute packaging, but the combination of gradle plus custom proovy plugins have proved beyond our understanding.

Our plan B is to use maven to build the missing packages, but the plugins are missing from maven central, and so we cannot declare them as dependencies of these packaging projects.

Downloading things manually is obviously not the solution, maven and deb/rpm have solved the distribution problems for us, however elasticsearch has not included plugins in that solution.

[dadoonet (David Pilato)]

My guess is that you have to build then the plugins yourself?

The source code is https://github.com/elastic/elasticsearch/tree/master/plugins.

gradle plugins:assemble builds them.

Would that work for you?

[Graham_Leggett]

It has not worked to date, no.

(Obviously download source code, manually build it, manually extract plugins and then manually inject into local maven repo to manually build debian packages is worse than manually download plugins, manually inject into local maven repo to manually build debian packages out of them, and then do this for every elasticsearch release)

We have tried extending the gradle build so that it produces deb/rpm files in addition to the zip file it produces now using the com.netflix.nebula:gradle-ospackage-plugin:3.4.0 plugin that is used now for packaging, but we've got nowhere. Elasticsearch is built using what appears to be pure gradle, and the com.netflix.nebula:gradle-ospackage-plugin:3.4.0 plugin is able to find artifacts and include them in the package. The plugins however are built using a set of custom plugins listed below. It is not clear how to extract the artifact and dependencies from these plugins in order to insert them into the debian/rpm packaging.

-rw-r--r--  1 minfrin  staff   3988 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginBuildPlugin.groovy
-rw-r--r--  1 minfrin  staff   1429 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginPropertiesExtension.groovy
-rw-r--r--  1 minfrin  staff   2761 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginPropertiesTask.groovy
-rw-r--r--  1 minfrin  staff  13415 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginBuildPlugin.groovy
-rw-r--r--  1 minfrin  staff   2073 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesExtension.groovy
-rw-r--r--  1 minfrin  staff   3624 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesTask.groovy

[Graham_Leggett]

I see that InstallPluginCommand.java contains code to download plugins from maven central, however with maven central now no longer hosting elasticsearch plugins, this code would be broken:

/** Returns the url for an elasticsearch plugin in maven. */
private String getMavenUrl(Terminal terminal, String[] coordinates, String platform) throws IOException {
    final String groupId = coordinates[0].replace(".", "/");
    final String artifactId = coordinates[1];
    final String version = coordinates[2];
    final String baseUrl = String.format(Locale.ROOT, "https://repo1.maven.org/maven2/%s/%s/%s", groupId, artifactId, version);
    final String platformUrl = String.format(Locale.ROOT, "%s/%s-%s-%s.zip", baseUrl, artifactId, platform, version);
    if (urlExists(terminal, platformUrl)) {
        return platformUrl;
    }
    return String.format(Locale.ROOT, "%s/%s-%s.zip", baseUrl, artifactId, version);
}

[dadoonet (David Pilato)]

No this code is still useful for the community who is building plugins and often distribute them on Maven central.

[Graham_Leggett]

Can you confirm where elasticsearch publishes the plugins if not maven central?

Reverse engineering the install code show the following URL prefix, but all the directory indexing is switched off.

https://artifacts.elastic.co/downloads/elasticsearch-plugins/analysis-icu

The docs tell you that it is possible to install a plugin from a local file or URL, but there is nothing to show where the URL is:

https://www.elastic.co/guide/en/elasticsearch/plugins/6.2/plugin-management-custom-url.html

[dadoonet (David Pilato)]

The documentation of each plugins tell you that

[s1monw (Simon Willnauer)]

Hey @Graham_Leggett I read through the linked issues and this topic. I must admit I don't necessarily understand the issues. On the one hand there are some issues with RPMs and plugins but we also discussing why plugins are not in maven central. I feel that there is some kind of disconnect so I will try to sum up some thing to clarify and ask some questions, hope that is ok.

Regarding Maven

We publish maven artifacts for historic reasons. We have a Java client that depends on elasticsearch core / server for request and response objects and serialization. Users of these clients have all necessary plugins in maven central that are required to run these clients. Our server distribution is not in maven central, it's distributed as a zip/tar.gz/rpm etc. from our download mirrors. If there is a plugin missing please let us know. To my knowledge there is non.

Regarding RPM

RPMs only contain the server and no plugins. If a plugin needs to be installed you can either install it from our mirrors with a URL pointing to it. (that is the default). We also allow you to fetch it from maven but that is not the official way and only a convenience for 3rd party engineers. The 3rd way is to install the plugin from a node-local directory or a NFS shard which allows you to download the plugins from our mirrors and place it there.

Given that's not acceptable for you to download from the internet directly which I understand, why is the 3rd option (local install) not an option for you? Please don't tell me it's stupid or anything like this. I want to understand what the issue is that prevents you doing it so we can fix it. I think we have a large userbase that uses RPMs and it's seems to be manageable from my perspective. I'd really like to understand the root cause.

simon