This makes it difficult to inject elasticsearch artifacts into maven projects.
I raised this as a bug at https://github.com/elastic/elasticsearch/issues/28765, however this bug was closed pending discussion, which has confused me. Is there an intention to keep publishing artifacts to maven central, or is the intention to stop doing so? If the elasticsearch project doesn't plan on publishing artifacts to maven central moving forward, what repository can I source the elasticsearch artifacts and plugins from?
Elasticsearch publishes debian and rpm packages for the main elasticsearch server, however none of the plugins have been packaged. We operate in an environment which offers a service level agreement, and so a best effort install process does not work for us.
We tried to contribute packaging, but the combination of gradle plus custom proovy plugins have proved beyond our understanding.
Our plan B is to use maven to build the missing packages, but the plugins are missing from maven central, and so we cannot declare them as dependencies of these packaging projects.
Downloading things manually is obviously not the solution, maven and deb/rpm have solved the distribution problems for us, however elasticsearch has not included plugins in that solution.
(Obviously download source code, manually build it, manually extract plugins and then manually inject into local maven repo to manually build debian packages is worse than manually download plugins, manually inject into local maven repo to manually build debian packages out of them, and then do this for every elasticsearch release)
We have tried extending the gradle build so that it produces deb/rpm files in addition to the zip file it produces now using the com.netflix.nebula:gradle-ospackage-plugin:3.4.0 plugin that is used now for packaging, but we've got nowhere. Elasticsearch is built using what appears to be pure gradle, and the com.netflix.nebula:gradle-ospackage-plugin:3.4.0 plugin is able to find artifacts and include them in the package. The plugins however are built using a set of custom plugins listed below. It is not clear how to extract the artifact and dependencies from these plugins in order to insert them into the debian/rpm packaging.
-rw-r--r-- 1 minfrin staff 3988 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginBuildPlugin.groovy
-rw-r--r-- 1 minfrin staff 1429 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginPropertiesExtension.groovy
-rw-r--r-- 1 minfrin staff 2761 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/MetaPluginPropertiesTask.groovy
-rw-r--r-- 1 minfrin staff 13415 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginBuildPlugin.groovy
-rw-r--r-- 1 minfrin staff 2073 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesExtension.groovy
-rw-r--r-- 1 minfrin staff 3624 Feb 19 19:14 buildSrc/src/main/groovy/org/elasticsearch/gradle/plugin/PluginPropertiesTask.groovy
I see that InstallPluginCommand.java contains code to download plugins from maven central, however with maven central now no longer hosting elasticsearch plugins, this code would be broken:
/** Returns the url for an elasticsearch plugin in maven. */
private String getMavenUrl(Terminal terminal, String[] coordinates, String platform) throws IOException {
final String groupId = coordinates[0].replace(".", "/");
final String artifactId = coordinates[1];
final String version = coordinates[2];
final String baseUrl = String.format(Locale.ROOT, "https://repo1.maven.org/maven2/%s/%s/%s", groupId, artifactId, version);
final String platformUrl = String.format(Locale.ROOT, "%s/%s-%s-%s.zip", baseUrl, artifactId, platform, version);
if (urlExists(terminal, platformUrl)) {
return platformUrl;
}
return String.format(Locale.ROOT, "%s/%s-%s.zip", baseUrl, artifactId, version);
}
Hey @Graham_Leggett I read through the linked issues and this topic. I must admit I don't necessarily understand the issues. On the one hand there are some issues with RPMs and plugins but we also discussing why plugins are not in maven central. I feel that there is some kind of disconnect so I will try to sum up some thing to clarify and ask some questions, hope that is ok.
Regarding Maven
We publish maven artifacts for historic reasons. We have a Java client that depends on elasticsearch core / server for request and response objects and serialization. Users of these clients have all necessary plugins in maven central that are required to run these clients. Our server distribution is not in maven central, it's distributed as a zip/tar.gz/rpm etc. from our download mirrors. If there is a plugin missing please let us know. To my knowledge there is non.
Regarding RPM
RPMs only contain the server and no plugins. If a plugin needs to be installed you can either install it from our mirrors with a URL pointing to it. (that is the default). We also allow you to fetch it from maven but that is not the official way and only a convenience for 3rd party engineers. The 3rd way is to install the plugin from a node-local directory or a NFS shard which allows you to download the plugins from our mirrors and place it there.
Given that's not acceptable for you to download from the internet directly which I understand, why is the 3rd option (local install) not an option for you? Please don't tell me it's stupid or anything like this. I want to understand what the issue is that prevents you doing it so we can fix it. I think we have a large userbase that uses RPMs and it's seems to be manageable from my perspective. I'd really like to understand the root cause.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.