Manual zip uploads to Github disabled

jprante · December 12, 2012, 1:01am

Hi,

any suggestions how to upload plugin zips and binaries to Github now?

I would suggest using
https://raw.github.com///master/downloads/-.zip
instead.

Thoughts?

Jörg

--

Ivan · December 12, 2012, 1:26am

I was about to post the exact same thing! Elasticsearch plugins were the
first thing I thought of when I read it.

Is the second link unaffected? I would assume everything marked "download",
no matter what the URL, will go away.

--
Ivan

On Tue, Dec 11, 2012 at 5:01 PM, Jörg Prante joergprante@gmail.com wrote:

Hi,

any suggestions how to upload plugin zips and binaries to Github now?

Goodbye, Uploads - The GitHub Blog

I would suggest using https://raw.github.com///master/downloads/-.zip
instead.

Thoughts?

Jörg

--

--

Ivan · December 12, 2012, 1:44am

Source control can handle more than source, so "raw" links should work.

This would require a change to PluginManager and whoever wants to install a
plugin via GitHub would need to upgrade. At least file based installs
should work.

BTW, file-based installs on Windows is a pain. Just had to throw that in.

Cheers,

Ivan

On Tue, Dec 11, 2012 at 5:26 PM, Ivan Brusic ivan@brusic.com wrote:

post the exact same thing! ElasticSe

--

jprante · December 12, 2012, 9:50am

The plugin service is only a basic solution and is limited when it comes to
dependency resolution. It relies on a single bundle, a zip archive, with
all dependencies included. Plugins are separated from each other and must
not have conflicting dependencies.

I would like to work on a more advanced implementation a bit, so that ES
nodes could be able to resolve dependencies from Maven repos when they
start up.

Pros:

Maven repos and dependencies would have to be declared in YAML in the
elasticsearch config
ES nodes could resolve the latest plugin versions at startup time
automatically, just by reusing the Maven artifact resolving mechanism
plugins could be scoped with the help of public/private plugins repo
config
plugins could share dependencies
the plugin distributions would be smaller, they would no longer need to
be bundled into a zip for download and install
external plugin resources could be maintained outside the plugin which
helps a lot when sharing code and data

Cons:

dependency resolution may take some extra time and space for collecting
artifacts into the local repo
if plugin resolution failures or conflicts occur, tackling them might
need extra Maven repo knowledge

There is a project out there, the Shrinkwrap resolver

that could help a lot. It's Apache License.

Cheers,

Jörg

On Wednesday, December 12, 2012 2:44:53 AM UTC+1, Ivan Brusic wrote:

Source control can handle more than source, so "raw" links should work.

This would require a change to PluginManager and whoever wants to install
a plugin via GitHub would need to upgrade. At least file based installs
should work.

BTW, file-based installs on Windows is a pain. Just had to throw that in.

Cheers,

Ivan

On Tue, Dec 11, 2012 at 5:26 PM, Ivan Brusic <iv...@brusic.com<javascript:>

wrote:

post the exact same thing! ElasticSe

--

Daniel_Maher_3 · December 12, 2012, 1:24pm

On 2012-12-12 2:26 AM, Ivan Brusic wrote:

I was about to post the exact same thing! Elasticsearch plugins were the
first thing I thought of when I read it.

Could we not just switch to using the "downloadable source code
archives" ? That functionality is not disappearing.

As an added bonus, if you use git tags in your repo, there will be .zip
and .tar.gz archives available automatically for your tagged release.
This makes handling versions pretty straightforward.

--
dan (phrawzty).
mozilla webops; european outpost.

--

dadoonet · December 12, 2012, 1:58pm

Sure, but the .zip and .tar.gz only contain source code, not the artifact zip
file containing the plugin jar AND its dependencies.
My 2 cents

That said, we perhaps push theses artifacts to the maven central repo and use
nexus from sonatype as the download service?

Best (IMHO), I wrote some days ago an idea of building a ES Store (like the APP
Store).
It's perhaps time to do it.

The bin/plugin command will help to display a list of available plugins
(depending on the current version of elasticsearch) and will help to download
it.

What do you think?
I will love to work on that (if it can be useful).

David.

Le 12 décembre 2012 à 14:24, Daniel Maher dmaher@mozilla.com a écrit :

On 2012-12-12 2:26 AM, Ivan Brusic wrote:

I was about to post the exact same thing! Elasticsearch plugins were the
first thing I thought of when I read it.

Could we not just switch to using the "downloadable source code
archives" ? That functionality is not disappearing.

As an added bonus, if you use git tags in your repo, there will be .zip
and .tar.gz archives available automatically for your tagged release.
This makes handling versions pretty straightforward.

--
dan (phrawzty).
mozilla webops; european outpost.

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

ferhatsb · December 12, 2012, 3:01pm

A plugin store can be interesting, it can be useful though it should be
more then a store. It should be also a repository.

From my point of view it is useful. David, we can also help to build it if
required or we can build it completely and share ideas here to make it
better.

Mind strom mode on (Categorized plugin data model,version tags, search! :),
github login, S3 hosted binaries, github api integration with activity
graphs, site links.)

Best,

Ferhat
www.searchbox.io

On Wednesday, December 12, 2012 3:58:28 PM UTC+2, David Pilato wrote:

Sure, but the .zip and .tar.gz only contain source code, not the
artifact zip file containing the plugin jar AND its dependencies.
My 2 cents

That said, we perhaps push theses artifacts to the maven central repo
and use nexus from sonatype as the download service?

Best (IMHO), I wrote some days ago an idea of building a ES Store (like
the APP Store).
It's perhaps time to do it.

The bin/plugin command will help to display a list of available plugins
(depending on the current version of elasticsearch) and will help to
download it.

What do you think?
I will love to work on that (if it can be useful).

David.

Le 12 décembre 2012 à 14:24, Daniel Maher <dma...@mozilla.com<javascript:>>
a écrit :

On 2012-12-12 2:26 AM, Ivan Brusic wrote:

I was about to post the exact same thing! Elasticsearch plugins were
the
first thing I thought of when I read it.

Could we not just switch to using the "downloadable source code
archives" ? That functionality is not disappearing.

As an added bonus, if you use git tags in your repo, there will be .zip
and .tar.gz archives available automatically for your tagged release.
This makes handling versions pretty straightforward.

--
dan (phrawzty).
mozilla webops; european outpost.

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Daniel_Maher_3 · December 12, 2012, 3:03pm

On 2012-12-12 4:01 PM, ferhatsb wrote:

A plugin store can be interesting, it can be useful though it should be
more then a store. It should be also a repository.

I'll just leave this here : http://gitlabhq.com/

--
dan (phrawzty).
mozilla webops; european outpost.

--

Ivan · December 12, 2012, 3:46pm

Part of the beauty of having the plugins as Github downloads is that the
barrier of entry for anyone to distribute an ES plugin was quite low.
Anyone that writes a plugin can distribute it easily. I am afraid that
every other solution would not be as easy and will hinder contributions.

A plugin store will greatly help discoverability. Currently, plugins are
listed on a page on the ES website, but this page is rarely update and the
the frequency of plugins being contributed is increasing. One-stop shopping
would be nice.

Cheers,

Ivan

On Wed, Dec 12, 2012 at 7:03 AM, Daniel Maher dmaher@mozilla.com wrote:

On 2012-12-12 4:01 PM, ferhatsb wrote:

A plugin store can be interesting, it can be useful though it should be
more then a store. It should be also a repository.

I'll just leave this here : http://gitlabhq.com/

--
dan (phrawzty).
mozilla webops; european outpost.

--

--

jprante · December 12, 2012, 7:36pm

Hi David,

how about moving all plugins for the ES app store to a new Maven artifact
groupID org.elasticsearch.apps ? With sub groups
org.elasticsearch.apps.xyz? So all these artifacts could be filtered for
being listed by a new tool for app searching/browsing, maybe bin/apps ?

My contribution would be code for the Maven dependency resolver in the
internal ES node.

Maven central or Nexus Sonatype could be the default setting, just like mvn
is behaving, but adding/removing repos should be possible. Note, I have a
Maven repo on github but I also think of company deployments in their
intranets with a private repo.

Cheers,

Jörg

--

dadoonet · December 12, 2012, 9:10pm

Hey Jörg,

It could be nice but that means that every contributor must be able to deploy its artefacts in sonatype repo under the groupId org.es.apps.xyz and I don't think it's possible.

As I maintain the plugin list on es.org github repo, I probably can maintain it in a JSon file available on es.org website and only make "links" to the sonatype repo, whatever the groupId is.

Something like :
{
"plugins":[
{
"name":"JDBC river",
"author":"Jorg Prante",
"groupId":"org.jorg",
"artifactId":"jdbc-river",
"versions":[
{
"version":"0.1",
"esversion":"[0.19,)",
"url":"http://nexus.sonatype.org/..." // or a repo name ???
}
], ...
}
]
}

That way, any company will also be able to provide its own plugins files.

What do you think?

--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 déc. 2012 à 20:36, Jörg Prante joergprante@gmail.com a écrit :

Hi David,

how about moving all plugins for the ES app store to a new Maven artifact groupID org.elasticsearch.apps ? With sub groups org.elasticsearch.apps.xyz? So all these artifacts could be filtered for being listed by a new tool for app searching/browsing, maybe bin/apps ?

My contribution would be code for the Maven dependency resolver in the internal ES node.

Maven central or Nexus Sonatype could be the default setting, just like mvn is behaving, but adding/removing repos should be possible. Note, I have a Maven repo on github but I also think of company deployments in their intranets with a private repo.

Cheers,

Jörg

--

jprante · December 12, 2012, 11:35pm

Hi David,

yes, it's better not to restrict to some groupId. Let the user configure
and decide!

Because it's so much fun, I started coding an ES Apps resolver service
prototype, based upon the PluginService

For testing purpose only, I found an
old org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven
Central

Interim results: resolving works, downloading works, repo creating works
like a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.

Now, still some fun parts to do, config stuff, and aligning the resolver
with the codepoints where ES initializes the plugins.

More later, cheers to all!

Jörg

    String mavenSettingsFile = settings.get("apps.settings",

"config/settings-es.xml");
String[] defaultAppClasses =
settings.getAsArray("apps.dependencies");
if (defaultAppClasses.length > 0) {
MavenDependency[] defaultDeps = new
MavenDependency[defaultAppClasses.length];
for (int i = 0; i < defaultAppClasses.length; i++) {
logger.info("adding dependency {}", defaultAppClasses[i]);
defaultDeps[i] =
MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,
false);
}

        MavenResolvedArtifact[] artifacts = Maven.configureResolver()
                .fromFile(mavenSettingsFile)
                .addDependencies(defaultDeps)
                .resolve()
                .withTransitivity()
                .asResolvedArtifact();

        for (MavenResolvedArtifact artifact : artifacts) {
            logger.info("found artifact {} {} {}",
                    artifact.getCoordinate().getGroupId(),
                    artifact.getCoordinate().getArtifactId(),
                    artifact.getCoordinate().getVersion());
        }
    }

Running org.elasticsearch.apps.AppResolverServiceTests
Configuring TestNG with:
org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445
[2012-12-13 00:10:40,876][INFO ][org.elasticsearch.apps ] [Terminus]
adding dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10
[2012-12-13 00:10:42,227][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.elasticsearch elasticsearch 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-core 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-analyzers 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-memory 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-queries 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact jakarta-regexp jakarta-regexp 1.4
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-highlighter 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact commons-io commons-io 1.2
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact com.rabbitmq amqp-client 2.5.1
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact commons-cli commons-cli 1.1

--

Richard_Louapre · December 13, 2012, 12:48pm

Hello,

Would it make sense to look at OSGI for plugins install, update?

Thanks,
Richard.

On Wednesday, December 12, 2012 6:35:21 PM UTC-5, Jörg Prante wrote:

Hi David,

yes, it's better not to restrict to some groupId. Let the user configure
and decide!

Because it's so much fun, I started coding an ES Apps resolver service
prototype, based upon the PluginService

https://github.com/jprante/elasticsearch-apps

For testing purpose only, I found an
old org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven
Central

Interim results: resolving works, downloading works, repo creating works
like a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.

Now, still some fun parts to do, config stuff, and aligning the resolver
with the codepoints where ES initializes the plugins.

More later, cheers to all!

Jörg
    String mavenSettingsFile = settings.get("apps.settings", 
"config/settings-es.xml");
String defaultAppClasses =
settings.getAsArray("apps.dependencies");
if (defaultAppClasses.length > 0) {
MavenDependency defaultDeps = new
MavenDependency[defaultAppClasses.length];
for (int i = 0; i < defaultAppClasses.length; i++) {
logger.info("adding dependency {}", defaultAppClasses[i]);
defaultDeps[i] =
MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,
false);
}
        MavenResolvedArtifact[] artifacts = Maven.configureResolver()
                .fromFile(mavenSettingsFile)
                .addDependencies(defaultDeps)
                .resolve()
                .withTransitivity()
                .asResolvedArtifact();

        for (MavenResolvedArtifact artifact : artifacts) {
            logger.info("found artifact {} {} {}",
                    artifact.getCoordinate().getGroupId(),
                    artifact.getCoordinate().getArtifactId(),
                    artifact.getCoordinate().getVersion());
        }
    }
Running org.elasticsearch.apps.AppResolverServiceTests
Configuring TestNG with:
org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445
[2012-12-13 00:10:40,876][INFO ][org.elasticsearch.apps ] [Terminus]
adding dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10
[2012-12-13 00:10:42,227][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.elasticsearch elasticsearch 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-core 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-analyzers 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-memory 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-queries 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact jakarta-regexp jakarta-regexp 1.4
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact org.apache.lucene lucene-highlighter 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact commons-io commons-io 1.2
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact com.rabbitmq amqp-client 2.5.1
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus]
found artifact commons-cli commons-cli 1.1

--

dadoonet · December 13, 2012, 2:37pm

Hi Jörg,

That's funny for sure.
So your idea is to only push in sonatype repo the pom and jar files (and not the
assembly - zip - itself).
Nice!

The main problem I can see here is that dependencies are resolved using the
pom.xml file. It's a problem because we don't need all libs when we use the
plugin.

For the plugin you tested, in the final ZIP file, you won't have elasticsearch
core lib and its dependencies.
If you look at

github.com

elastic/elasticsearch-river-rabbitmq/blob/master/src/main/assemblies/plugin.xml

<?xml version="1.0"?>
<assembly>
    <id>plugin</id>
    <formats>
        <format>zip</format>
    </formats>
    <includeBaseDirectory>false</includeBaseDirectory>
    <dependencySets>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <useTransitiveFiltering>true</useTransitiveFiltering>
            <excludes>
                <exclude>org.elasticsearch:elasticsearch</exclude>
            </excludes>
        </dependencySet>
        <dependencySet>
            <outputDirectory>/</outputDirectory>
            <useProjectArtifact>true</useProjectArtifact>
            <useTransitiveFiltering>true</useTransitiveFiltering>

This file has been truncated. show original

file, you will see that org.elasticsearch:elasticsearch and its dependencies is
excluded.

Because, in pom.xml, it's defined as follow:

     <dependency>
         <groupId>org.elasticsearch </groupId>
         <artifactId>elasticsearch </artifactId>
         <version>${elasticsearch.version} </version>
         <scope>compile </scope>
     </dependency>

Perhaps, we should define the scope for elasticsearch only as "provided" instead
of "compile". I don't know what will be the effect for your ES apps resolver
project.

About what Richard said with OSGi, I think it's a little more complicated as,
AFAIK, Elasticsearch doesn't have OSGi capabilities yet.
But, being able to add plugins on a running ES instance is a nice idea for sure.

Perhaps we can start with a simplier approach and only try to fix first the
problem we have with Github downloads.

Then, let's wait for Shay and the ES Team thoughts abour OSGi in ES.

What do you think?

David

Le 13 décembre 2012 à 00:35, "Jörg Prante" joergprante@gmail.com a écrit :

Hi David,

yes, it's better not to restrict to some groupId. Let the user configure and
decide!

Because it's so much fun, I started coding an ES Apps resolver service
prototype, based upon the PluginService

https://github.com/jprante/elasticsearch-apps

For testing purpose only, I found an old
org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven Central

Interim results: resolving works, downloading works, repo creating works like
a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.

Now, still some fun parts to do, config stuff, and aligning the resolver with
the codepoints where ES initializes the plugins.

More later, cheers to all!

Jörg
     String mavenSettingsFile = settings.get("apps.settings",
"config/settings-es.xml");
String defaultAppClasses =
settings.getAsArray("apps.dependencies");
if (defaultAppClasses.length > 0) {
MavenDependency defaultDeps = new
MavenDependency[defaultAppClasses.length];
for (int i = 0; i < defaultAppClasses.length; i++) {
logger.info("adding dependency {}", defaultAppClasses[i]);
defaultDeps[i] =
MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,
false);
}
         MavenResolvedArtifact[] artifacts = Maven.configureResolver()
                 .fromFile(mavenSettingsFile)
                 .addDependencies(defaultDeps)
                 .resolve()
                 .withTransitivity()
                 .asResolvedArtifact();

         for (MavenResolvedArtifact artifact : artifacts) {
             logger.info("found artifact {} {} {}",
                     artifact.getCoordinate().getGroupId(),
                     artifact.getCoordinate().getArtifactId(),
                     artifact.getCoordinate().getVersion());
         }
     }
Running org.elasticsearch.apps.AppResolverServiceTests
Configuring TestNG with:
org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445
[2012-12-13 00:10:40,876][INFO ][org.elasticsearch.apps ] [Terminus] adding
dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10
[2012-12-13 00:10:42,227][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.elasticsearch elasticsearch 0.17.10
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.apache.lucene lucene-core 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.apache.lucene lucene-analyzers 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.apache.lucene lucene-memory 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.apache.lucene lucene-queries 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact jakarta-regexp jakarta-regexp 1.4
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact org.apache.lucene lucene-highlighter 3.4.0
[2012-12-13 00:10:42,228][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact commons-io commons-io 1.2
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact com.rabbitmq amqp-client 2.5.1
[2012-12-13 00:10:42,229][INFO ][org.elasticsearch.apps ] [Terminus] found
artifact commons-cli commons-cli 1.1

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

jprante · December 13, 2012, 3:39pm

David,

that was just half-ready code, without the exclusions. Yes, the app
resolver is able to exclude elasticsearch itself, I just went to get some
sleep

Of course, JBoss Shrinkwrap can do what Maven can, so excluding
dependencies is not a problem.

The code - still a bit poor - is now looking like this:

    String mavenSettingsFile = settings.get("apps.settings",

"config/settings-es.xml");
final MavenDependencyExclusion exclusion = MavenDependencies
.createExclusion("org.elasticsearch:elasticsearch");
MavenResolvedArtifact[] artifacts = null;
String[] defaultAppClasses =
settings.getAsArray("apps.dependencies");
if (defaultAppClasses.length > 0) {
MavenDependency[] defaultDeps = new
MavenDependency[defaultAppClasses.length];
for (int i = 0; i < defaultAppClasses.length; i++) {
logger.info("adding dependency {}", defaultAppClasses[i]);
defaultDeps[i] =
MavenDependencies.createDependency(defaultAppClasses[i],
ScopeType.COMPILE, false, exclusion);
}
artifacts = Maven.configureResolver()
.fromFile(mavenSettingsFile)
.addDependencies(defaultDeps)
.resolve()
.withMavenCentralRepo(true)
.withTransitivity()
.asResolvedArtifact();
}

It's still not complete. New parameters will be provided to
config/elasticsearch.yml to control the app resolving.

Also note, conf/settings-es.xml is a full functional Maven settings.xml,
with all your favorite Maven-specific customized repositories, and so on.

For site plugins, there will be just URLs to ZIP files for download, no
Maven artifacts.

I the app system, you should also be able to execute something like

bin/apps -update

and all your current plugins will be updated to the latest version!

So I am heading towards an app resolver class that is able to load all
plugins with dependency resolution, versioning, more information via REST
endpoints, and all the fancy stuff!

Cheers,

Jörg

--

Lukas_Vlcek1 · December 16, 2012, 8:50am

Hi Ivan,

I think what Jörg suggests will still allow this.

An plugin author can still prepare plugin distribution including all its
dependencies as a singlt file using shade (
Apache Maven Shade Plugin – Introduction) - btw, that is what
elasticsearch already uses to include its own dependencies into its JAR.
Then the author can upload this result artefact (plugin.jar) anywhere
on the internet (it can be github for example) and tell people how to add
maven repository (or probably instruct the elasticsearch-app on command
line?) to download and install it into ES node.

At least that is how I understand what Jörg is suggesting and started
working on.

Yes, there might be some extra step compared to current github plugin
distribution model but that model is going away anyway.

One very nice benefit is the fact that plugins can reuse common code and
declare dependencies on each other (and using Maven pom for this means we
do not have to reinvent the wheel). Going forward I do not think we can
avoid more developed/advanced plugin dependency model anyway. It will also
bring a lot of advantages. For example think of plugin that rely on other
plugins. Then the only think the end user has to think about is installing
the "top most" plugin. All the other plugins will be downloaded/updated
behind the scene.

Regards,
Lukas

On Wed, Dec 12, 2012 at 4:46 PM, Ivan Brusic ivan@brusic.com wrote:

Part of the beauty of having the plugins as Github downloads is that the
barrier of entry for anyone to distribute an ES plugin was quite low.
Anyone that writes a plugin can distribute it easily. I am afraid that
every other solution would not be as easy and will hinder contributions.

--

jprante · December 17, 2012, 8:59am

Exactly, I am experimenting with an app tool that can work with three types
of apps. "Legacy zip plugin" apps (as we know it, URL-based, without
maven-based auto update), "site zip plugin" apps (URL-based, without Java
classpath plugin declaration), and a new kind of "artifact apps" (a jar
artifact from an unzipped plugin that exist in an arbitrary Maven repo).

There is an extra step and more work when releasing legacy zip plugins I
think. Currently, plugin authors must create a special ES plugin formatted
zip package with the maven assembly and move it manually to a location
where it can be downloaded. But a Maven jar plugin author will be able to
create and deploy everything by a single command "mvn deploy", which is
less work, see
http://maven.apache.org/plugins/maven-deploy-plugin/usage.html

I will also give a documentation how a Maven repo can be maintained on
github, by deploying an ES app locally to a github repo, and committing it
to github, as an extra step. Certainly not a best-practice approach, Maven
central should be of course the preferred way.

By declaring the Maven artifacts in config/elasticsearch.yml as a list of
artifact coordinates (group ID, artifact ID, and version), ES apps could be
installed and maintained including dependencies and versioning.

Just a "bin/apps -update" could go through the list of apps installed,
modify the version descriptor in the coordinate, and run a dependency
resolution to download the latest version.

Even plugins that depend on other plugins would be easy to organize. Just
imagine an example: One author learns about Lucene 4, writes a simple
Lucene 4 codec with a little bit of ES integration, and publishes this as
an ES app via a Maven repo. It is neat, but not easy to use by everybody in
the community because it is not exposing REST endpoints, for example.
Another author adds this ES app as a dependency to his pom.xml, completes
the integration, writes REST endpoints for this plugin, adds more tools,
and publishes this also as an ES app via a maven repo.

Discoverability at remote places is a harder task. Some clever repo
browsing would be needed. Resolving artifacts in a repo works if a full
Maven coordinate is already known from configuration. As it looks now, all
candidate artifacts have to be downloaded into the local maven repo. Each
artifact must be opened in a loop, and a temporary classpath must be
examined for the es-plugin.properties resources, the ES plugin
identification. To limit the time-consuming traversal, I think of something
like "bin/app -discover ", so only a certain Maven group could be
examined. This can behave like a "one stop shop".

And maybe later, in an advanced app tool version in the future, the local
Maven repo index could automatically be indexed by ES itself to enable a
cool app search - hello, ES maven repo river?

Cheers,

Jörg

On Sunday, December 16, 2012 9:50:44 AM UTC+1, Lukáš Vlček wrote:

Hi Ivan,

I think what Jörg suggests will still allow this.

An plugin author can still prepare plugin distribution including all
its dependencies as a singlt file using shade (
Apache Maven Shade Plugin – Introduction) - btw, that is what
elasticsearch already uses to include its own dependencies into its JAR.

Then the author can upload this result artefact (plugin.jar) anywhere
on the internet (it can be github for example) and tell people how to add
maven repository (or probably instruct the elasticsearch-app on command
line?) to download and install it into ES node.

At least that is how I understand what Jörg is suggesting and started
working on.

Yes, there might be some extra step compared to current github plugin
distribution model but that model is going away anyway.

One very nice benefit is the fact that plugins can reuse common code and
declare dependencies on each other (and using Maven pom for this means we
do not have to reinvent the wheel). Going forward I do not think we can
avoid more developed/advanced plugin dependency model anyway. It will also
bring a lot of advantages. For example think of plugin that rely on other
plugins. Then the only think the end user has to think about is installing
the "top most" plugin. All the other plugins will be downloaded/updated
behind the scene.

Regards,
Lukas

On Wed, Dec 12, 2012 at 4:46 PM, Ivan Brusic <iv...@brusic.com<javascript:>

wrote:

Part of the beauty of having the plugins as Github downloads is that the
barrier of entry for anyone to distribute an ES plugin was quite low.
Anyone that writes a plugin can distribute it easily. I am afraid that
every other solution would not be as easy and will hinder contributions.

--