# Manual zip uploads to Github disabled

**URL:** <https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049>\
**Category:** Elasticsearch\
**Created:** [December 12, 2012, 1:01am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049 "2012-12-12T01:01:56Z")\
**Posts on this page:** 18\
**Page:** 1

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 12, 2012, 1:01am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/1 "2012-12-12T01:01:56Z")

</div>

Hi,

any suggestions how to upload plugin zips and binaries to Github now?

> **[Goodbye, Uploads](https://github.com/blog/1302-goodbye-uploads)**
>
> In addition to providing downloadable source code archives, GitHub previously allowed you to upload files (separate from the versioned files) in the repository, and make it available for download i...

I would suggest using  
[https://raw.github.com/](https://raw.github.com/)//master/downloads/-.zip  
instead.

Thoughts?

Jörg

--

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [December 12, 2012, 1:26am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/2 "2012-12-12T01:26:42Z")

</div>

I was about to post the exact same thing! Elasticsearch plugins were the  
first thing I thought of when I read it.

Is the second link unaffected? I would assume everything marked "download",  
no matter what the URL, will go away.

--  
Ivan

On Tue, Dec 11, 2012 at 5:01 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:

> Hi,
> 
> any suggestions how to upload plugin zips and binaries to Github now?
> 
> [Goodbye, Uploads - The GitHub Blog](https://github.com/blog/1302-goodbye-uploads)
> 
> I would suggest using [https://raw.github.com/](https://raw.github.com/)//master/downloads/-.zip  
> instead.
> 
> Thoughts?
> 
> Jörg
> 
> --

--

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [December 12, 2012, 1:44am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/3 "2012-12-12T01:44:53Z")

</div>

Source control can handle more than source, so "raw" links should work.

This would require a change to PluginManager and whoever wants to install a  
plugin via GitHub would need to upgrade. At least file based installs  
should work.

BTW, file-based installs on Windows is a pain. Just had to throw that in. 🙂

Cheers,

Ivan

On Tue, Dec 11, 2012 at 5:26 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:

> post the exact same thing! ElasticSe

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 12, 2012, 9:50am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/4 "2012-12-12T09:50:10Z")

</div>

The plugin service is only a basic solution and is limited when it comes to  
dependency resolution. It relies on a single bundle, a zip archive, with  
all dependencies included. Plugins are separated from each other and must  
not have conflicting dependencies.

I would like to work on a more advanced implementation a bit, so that ES  
nodes could be able to resolve dependencies from Maven repos when they  
start up.

Pros:

- Maven repos and dependencies would have to be declared in YAML in the  
elasticsearch config

- ES nodes could resolve the latest plugin versions at startup time  
automatically, just by reusing the Maven artifact resolving mechanism

- plugins could be scoped with the help of public/private plugins repo  
config

- plugins could share dependencies

- the plugin distributions would be smaller, they would no longer need to  
be bundled into a zip for download and install

- external plugin resources could be maintained outside the plugin which  
helps a lot when sharing code and data

Cons:

- dependency resolution may take some extra time and space for collecting  
artifacts into the local repo

- if plugin resolution failures or conflicts occur, tackling them might  
need extra Maven repo knowledge

There is a project out there, the Shrinkwrap resolver

> **[GitHub - shrinkwrap/resolver: ShrinkWrap Resolvers](https://github.com/shrinkwrap/resolver)**
>
> ShrinkWrap Resolvers. Contribute to shrinkwrap/resolver development by creating an account on GitHub.

that could help a lot. It's Apache License.

Cheers,

Jörg

On Wednesday, December 12, 2012 2:44:53 AM UTC+1, Ivan Brusic wrote:

> Source control can handle more than source, so "raw" links should work.
> 
> This would require a change to PluginManager and whoever wants to install  
> a plugin via GitHub would need to upgrade. At least file based installs  
> should work.
> 
> BTW, file-based installs on Windows is a pain. Just had to throw that in.  
> 🙂
> 
> Cheers,
> 
> Ivan
> 
> On Tue, Dec 11, 2012 at 5:26 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\<javascript:\>
> 
> > wrote:
> 
> > post the exact same thing! ElasticSe

--

---

<div class="post-metadata">

**Author:** ![Daniel\_Maher\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_maher_3/32/2014_2.png) [@Daniel\_Maher\_3](https://discuss.elastic.co/u/Daniel_Maher_3)\
**Post date:** [December 12, 2012, 1:24pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/5 "2012-12-12T13:24:16Z")

</div>

On 2012-12-12 2:26 AM, Ivan Brusic wrote:

> I was about to post the exact same thing! Elasticsearch plugins were the  
> first thing I thought of when I read it.

Could we not just switch to using the "downloadable source code  
archives" ? That functionality is not disappearing.

As an added bonus, if you use git tags in your repo, there will be .zip  
and .tar.gz archives available automatically for your tagged release.  
This makes handling versions pretty straightforward.

--  
dan (phrawzty).  
mozilla webops; european outpost.

--

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 12, 2012, 1:58pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/6 "2012-12-12T13:58:28Z")

</div>

Sure, but the .zip and .tar.gz only contain source code, not the artifact zip  
file containing the plugin jar AND its dependencies.  
My 2 cents

That said, we perhaps push theses artifacts to the maven central repo and use  
nexus from sonatype as the download service?

Best (IMHO), I wrote some days ago an idea of building a ES Store (like the APP  
Store).  
It's perhaps time to do it.

The bin/plugin command will help to display a list of available plugins  
(depending on the current version of elasticsearch) and will help to download  
it.

What do you think?  
I will love to work on that (if it can be useful).

David.

Le 12 décembre 2012 à 14:24, Daniel Maher [dmaher@mozilla.com](mailto:dmaher@mozilla.com) a écrit :

> On 2012-12-12 2:26 AM, Ivan Brusic wrote:
> 
> > I was about to post the exact same thing! Elasticsearch plugins were the  
> > first thing I thought of when I read it.
> 
> Could we not just switch to using the "downloadable source code  
> archives" ? That functionality is not disappearing.
> 
> As an added bonus, if you use git tags in your repo, there will be .zip  
> and .tar.gz archives available automatically for your tagged release.  
> This makes handling versions pretty straightforward.
> 
> --  
> dan (phrawzty).  
> mozilla webops; european outpost.
> 
> --

--  
David Pilato  
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)  
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

---

<div class="post-metadata">

**Author:** ![ferhatsb](https://avatars.discourse-cdn.com/v4/letter/f/977dab/32.png) [@ferhatsb](https://discuss.elastic.co/u/ferhatsb)\
**Post date:** [December 12, 2012, 3:01pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/7 "2012-12-12T15:01:41Z")

</div>

A plugin store can be interesting, it can be useful though it should be  
more then a store. It should be also a repository.

From my point of view it is useful. David, we can also help to build it if  
required or we can build it completely and share ideas here to make it  
better.

Mind strom mode on (Categorized plugin data model,version tags, search! :),  
github login, S3 hosted binaries, github api integration with activity  
graphs, site links.)

Best,

Ferhat  
[www.searchbox.io](http://www.searchbox.io)

On Wednesday, December 12, 2012 3:58:28 PM UTC+2, David Pilato wrote:

> Sure, but the .zip and .tar.gz only contain source code, not the  
> artifact zip file containing the plugin jar AND its dependencies.  
> My 2 cents
> 
> That said, we perhaps push theses artifacts to the maven central repo  
> and use nexus from sonatype as the download service?
> 
> Best (IMHO), I wrote some days ago an idea of building a ES Store (like  
> the APP Store).  
> It's perhaps time to do it.
> 
> The bin/plugin command will help to display a list of available plugins  
> (depending on the current version of elasticsearch) and will help to  
> download it.
> 
> What do you think?  
> I will love to work on that (if it can be useful).
> 
> David.
> 
> Le 12 décembre 2012 à 14:24, Daniel Maher \<[dma...@mozilla.com](mailto:dma...@mozilla.com)\<javascript:\>\>  
> a écrit :
> 
> > On 2012-12-12 2:26 AM, Ivan Brusic wrote:
> > 
> > > I was about to post the exact same thing! Elasticsearch plugins were  
> > > the  
> > > first thing I thought of when I read it.
> > 
> > Could we not just switch to using the "downloadable source code  
> > archives" ? That functionality is not disappearing.
> > 
> > As an added bonus, if you use git tags in your repo, there will be .zip  
> > and .tar.gz archives available automatically for your tagged release.  
> > This makes handling versions pretty straightforward.
> > 
> > --  
> > dan (phrawzty).  
> > mozilla webops; european outpost.
> > 
> > --
> 
> --  
> David Pilato  
> [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)  
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

---

<div class="post-metadata">

**Author:** ![Daniel\_Maher\_3](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/daniel_maher_3/32/2014_2.png) [@Daniel\_Maher\_3](https://discuss.elastic.co/u/Daniel_Maher_3)\
**Post date:** [December 12, 2012, 3:03pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/8 "2012-12-12T15:03:02Z")

</div>

On 2012-12-12 4:01 PM, ferhatsb wrote:

> A plugin store can be interesting, it can be useful though it should be  
> more then a store. It should be also a repository.

I'll just leave this here : [http://gitlabhq.com/](http://gitlabhq.com/)

--  
dan (phrawzty).  
mozilla webops; european outpost.

--

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [December 12, 2012, 3:46pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/9 "2012-12-12T15:46:54Z")

</div>

Part of the beauty of having the plugins as Github downloads is that the  
barrier of entry for anyone to distribute an ES plugin was quite low.  
Anyone that writes a plugin can distribute it easily. I am afraid that  
every other solution would not be as easy and will hinder contributions.

A plugin store will greatly help discoverability. Currently, plugins are  
listed on a page on the ES website, but this page is rarely update and the  
the frequency of plugins being contributed is increasing. One-stop shopping  
would be nice.

Cheers,

Ivan

On Wed, Dec 12, 2012 at 7:03 AM, Daniel Maher [dmaher@mozilla.com](mailto:dmaher@mozilla.com) wrote:

> On 2012-12-12 4:01 PM, ferhatsb wrote:
> 
> > A plugin store can be interesting, it can be useful though it should be  
> > more then a store. It should be also a repository.
> 
> I'll just leave this here : [http://gitlabhq.com/](http://gitlabhq.com/)
> 
> --  
> dan (phrawzty).  
> mozilla webops; european outpost.
> 
> --

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 12, 2012, 7:36pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/10 "2012-12-12T19:36:00Z")

</div>

Hi David,

how about moving all plugins for the ES app store to a new Maven artifact  
groupID org.elasticsearch.apps ? With sub groups  
org.elasticsearch.apps.xyz? So all these artifacts could be filtered for  
being listed by a new tool for app searching/browsing, maybe bin/apps ?

My contribution would be code for the Maven dependency resolver in the  
internal ES node.

Maven central or Nexus Sonatype could be the default setting, just like mvn  
is behaving, but adding/removing repos should be possible. Note, I have a  
Maven repo on github 🙂 but I also think of company deployments in their  
intranets with a private repo.

Cheers,

Jörg

--

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 12, 2012, 9:10pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/11 "2012-12-12T21:10:26Z")

</div>

Hey Jörg,

It could be nice but that means that every contributor must be able to deploy its artefacts in sonatype repo under the groupId org.es.apps.xyz and I don't think it's possible.

As I maintain the plugin list on es.org github repo, I probably can maintain it in a JSon file available on es.org website and only make "links" to the sonatype repo, whatever the groupId is.

Something like :  
{  
"plugins":[  
{  
"name":"JDBC river",  
"author":"Jorg Prante",  
"groupId":"org.jorg",  
"artifactId":"jdbc-river",  
"versions":[  
{  
"version":"0.1",  
"esversion":"[0.19,)",  
"url":"[http://nexus.sonatype.org/](http://nexus.sonatype.org/)..." // or a repo name ???  
}  
], ...  
}  
]  
}

That way, any company will also be able to provide its own plugins files.

What do you think?

--  
David 😉  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 déc. 2012 à 20:36, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) a écrit :

Hi David,

how about moving all plugins for the ES app store to a new Maven artifact groupID org.elasticsearch.apps ? With sub groups org.elasticsearch.apps.xyz? So all these artifacts could be filtered for being listed by a new tool for app searching/browsing, maybe bin/apps ?

My contribution would be code for the Maven dependency resolver in the internal ES node.

Maven central or Nexus Sonatype could be the default setting, just like mvn is behaving, but adding/removing repos should be possible. Note, I have a Maven repo on github 🙂 but I also think of company deployments in their intranets with a private repo.

Cheers,

## Jörg

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 12, 2012, 11:35pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/12 "2012-12-12T23:35:21Z")

</div>

Hi David,

yes, it's better not to restrict to some groupId. Let the user configure  
and decide!

Because it's so much fun, I started coding an ES Apps resolver service  
prototype, based upon the PluginService

> **[jprante/elasticsearch-apps](https://github.com/jprante/elasticsearch-apps)**
>
> elasticsearch-apps - Fun with ES apps! Integrating Maven artifact resolving into Elasticsearch

For testing purpose only, I found an  
old org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven  
Central

Interim results: resolving works, downloading works, repo creating works  
like a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.

Now, still some fun parts to do, config stuff, and aligning the resolver  
with the codepoints where ES initializes the plugins.

More later, cheers to all!

Jörg

```
    String mavenSettingsFile = settings.get("apps.settings", 

```

"config/settings-es.xml");  
String[] defaultAppClasses =  
settings.getAsArray("apps.dependencies");  
if (defaultAppClasses.length \> 0) {  
MavenDependency[] defaultDeps = new  
MavenDependency[defaultAppClasses.length];  
for (int i = 0; i \< defaultAppClasses.length; i++) {  
logger.info("adding dependency {}", defaultAppClasses[i]);  
defaultDeps[i] =  
MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,  
false);  
}

```
        MavenResolvedArtifact[] artifacts = Maven.configureResolver()
                .fromFile(mavenSettingsFile)
                .addDependencies(defaultDeps)
                .resolve()
                .withTransitivity()
                .asResolvedArtifact();

        for (MavenResolvedArtifact artifact : artifacts) {
            logger.info("found artifact {} {} {}",
                    artifact.getCoordinate().getGroupId(),
                    artifact.getCoordinate().getArtifactId(),
                    artifact.getCoordinate().getVersion());
        }
    }

```

Running org.elasticsearch.apps.AppResolverServiceTests  
Configuring TestNG with:  
org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445  
[2012-12-13 00:10:40,876][INFO][org.elasticsearch.apps] [Terminus]  
adding dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10  
[2012-12-13 00:10:42,227][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.elasticsearch elasticsearch 0.17.10  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.apache.lucene lucene-core 3.4.0  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.apache.lucene lucene-analyzers 3.4.0  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.apache.lucene lucene-memory 3.4.0  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.apache.lucene lucene-queries 3.4.0  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact jakarta-regexp jakarta-regexp 1.4  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact org.apache.lucene lucene-highlighter 3.4.0  
[2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
found artifact commons-io commons-io 1.2  
[2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus]  
found artifact com.rabbitmq amqp-client 2.5.1  
[2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus]  
found artifact commons-cli commons-cli 1.1

--

---

<div class="post-metadata">

**Author:** ![Richard\_Louapre](https://avatars.discourse-cdn.com/v4/letter/r/e19adc/32.png) [@Richard\_Louapre](https://discuss.elastic.co/u/Richard_Louapre)\
**Post date:** [December 13, 2012, 12:48pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/13 "2012-12-13T12:48:18Z")

</div>

Hello,

Would it make sense to look at OSGI for plugins install, update?

Thanks,  
Richard.

On Wednesday, December 12, 2012 6:35:21 PM UTC-5, Jörg Prante wrote:

> Hi David,
> 
> yes, it's better not to restrict to some groupId. Let the user configure  
> and decide!
> 
> Because it's so much fun, I started coding an ES Apps resolver service  
> prototype, based upon the PluginService
> 
> [https://github.com/jprante/elasticsearch-apps](https://github.com/jprante/elasticsearch-apps)
> 
> For testing purpose only, I found an  
> old org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven  
> Central
> 
> Interim results: resolving works, downloading works, repo creating works  
> like a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.
> 
> Now, still some fun parts to do, config stuff, and aligning the resolver  
> with the codepoints where ES initializes the plugins.
> 
> More later, cheers to all!
> 
> Jörg
> 
> ```
> String mavenSettingsFile = settings.get("apps.settings", 
> 
> ```
> 
> "config/settings-es.xml");  
> String defaultAppClasses =  
> settings.getAsArray("apps.dependencies");  
> if (defaultAppClasses.length \> 0) {  
> MavenDependency defaultDeps = new  
> MavenDependency[defaultAppClasses.length];  
> for (int i = 0; i \< defaultAppClasses.length; i++) {  
> logger.info("adding dependency {}", defaultAppClasses[i]);  
> defaultDeps[i] =  
> MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,  
> false);  
> }
> 
> ```
> MavenResolvedArtifact[] artifacts = Maven.configureResolver()
> .fromFile(mavenSettingsFile)
> .addDependencies(defaultDeps)
> .resolve()
> .withTransitivity()
> .asResolvedArtifact();
> 
> for (MavenResolvedArtifact artifact : artifacts) {
> logger.info("found artifact {} {} {}",
> artifact.getCoordinate().getGroupId(),
> artifact.getCoordinate().getArtifactId(),
> artifact.getCoordinate().getVersion());
> }
> }
> 
> ```
> 
> Running org.elasticsearch.apps.AppResolverServiceTests  
> Configuring TestNG with:  
> org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445  
> [2012-12-13 00:10:40,876][INFO][org.elasticsearch.apps] [Terminus]  
> adding dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10  
> [2012-12-13 00:10:42,227][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.elasticsearch elasticsearch 0.17.10  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.apache.lucene lucene-core 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.apache.lucene lucene-analyzers 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.apache.lucene lucene-memory 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.apache.lucene lucene-queries 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact jakarta-regexp jakarta-regexp 1.4  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact org.apache.lucene lucene-highlighter 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact commons-io commons-io 1.2  
> [2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact com.rabbitmq amqp-client 2.5.1  
> [2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus]  
> found artifact commons-cli commons-cli 1.1

--

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 13, 2012, 2:37pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/14 "2012-12-13T14:37:35Z")

</div>

Hi Jörg,

That's funny for sure.  
So your idea is to only push in sonatype repo the pom and jar files (and not the  
assembly - zip - itself).  
Nice!

The main problem I can see here is that dependencies are resolved using the  
pom.xml file. It's a problem because we don't need all libs when we use the  
plugin.

For the plugin you tested, in the final ZIP file, you won't have elasticsearch  
core lib and its dependencies.  
If you look at

> <https://github.com/elastic/elasticsearch-river-rabbitmq/blob/master/src/main/assemblies/plugin.xml>

file, you will see that org.elasticsearch:elasticsearch and its dependencies is  
excluded.

Because, in pom.xml, it's defined as follow:

```
     <dependency>
         <groupId>org.elasticsearch </groupId>
         <artifactId>elasticsearch </artifactId>
         <version>${elasticsearch.version} </version>
         <scope>compile </scope>
     </dependency>

```

Perhaps, we should define the scope for elasticsearch only as "provided" instead  
of "compile". I don't know what will be the effect for your ES apps resolver  
project.

About what Richard said with OSGi, I think it's a little more complicated as,  
AFAIK, Elasticsearch doesn't have OSGi capabilities yet.  
But, being able to add plugins on a running ES instance is a nice idea for sure.

Perhaps we can start with a simplier approach and only try to fix first the  
problem we have with Github downloads.

Then, let's wait for Shay and the ES Team thoughts abour OSGi in ES.

What do you think?

David

Le 13 décembre 2012 à 00:35, "Jörg Prante" [joergprante@gmail.com](mailto:joergprante@gmail.com) a écrit :

> Hi David,
> 
> yes, it's better not to restrict to some groupId. Let the user configure and  
> decide!
> 
> Because it's so much fun, I started coding an ES Apps resolver service  
> prototype, based upon the PluginService
> 
> [https://github.com/jprante/elasticsearch-apps](https://github.com/jprante/elasticsearch-apps)
> 
> For testing purpose only, I found an old  
> org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10 plugin on Maven Central
> 
> Interim results: resolving works, downloading works, repo creating works like  
> a charme - thanks to the freaking awesome code of JBoss Shrinkwrap.
> 
> Now, still some fun parts to do, config stuff, and aligning the resolver with  
> the codepoints where ES initializes the plugins.
> 
> More later, cheers to all!
> 
> Jörg
> 
> ```
> String mavenSettingsFile = settings.get("apps.settings",
> 
> ```
> 
> "config/settings-es.xml");  
> String defaultAppClasses =  
> settings.getAsArray("apps.dependencies");  
> if (defaultAppClasses.length \> 0) {  
> MavenDependency defaultDeps = new  
> MavenDependency[defaultAppClasses.length];  
> for (int i = 0; i \< defaultAppClasses.length; i++) {  
> logger.info("adding dependency {}", defaultAppClasses[i]);  
> defaultDeps[i] =  
> MavenDependencies.createDependency(defaultAppClasses[i], ScopeType.COMPILE,  
> false);  
> }
> 
> ```
> MavenResolvedArtifact[] artifacts = Maven.configureResolver()
> .fromFile(mavenSettingsFile)
> .addDependencies(defaultDeps)
> .resolve()
> .withTransitivity()
> .asResolvedArtifact();
> 
> for (MavenResolvedArtifact artifact : artifacts) {
> logger.info("found artifact {} {} {}",
> artifact.getCoordinate().getGroupId(),
> artifact.getCoordinate().getArtifactId(),
> artifact.getCoordinate().getVersion());
> }
> }
> 
> ```
> 
> Running org.elasticsearch.apps.AppResolverServiceTests  
> Configuring TestNG with:  
> org.apache.maven.surefire.testng.conf.TestNG652Configurator@55d15445  
> [2012-12-13 00:10:40,876][INFO][org.elasticsearch.apps] [Terminus] adding  
> dependency org.elasticsearch:elasticsearch-river-rabbitmq:0.17.10  
> [2012-12-13 00:10:42,227][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.elasticsearch elasticsearch-river-rabbitmq 0.17.10  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.elasticsearch elasticsearch 0.17.10  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.apache.lucene lucene-core 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.apache.lucene lucene-analyzers 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.apache.lucene lucene-memory 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.apache.lucene lucene-queries 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact jakarta-regexp jakarta-regexp 1.4  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact org.apache.lucene lucene-highlighter 3.4.0  
> [2012-12-13 00:10:42,228][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact commons-io commons-io 1.2  
> [2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact com.rabbitmq amqp-client 2.5.1  
> [2012-12-13 00:10:42,229][INFO][org.elasticsearch.apps] [Terminus] found  
> artifact commons-cli commons-cli 1.1
> 
> --

--  
David Pilato  
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)  
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 13, 2012, 3:39pm UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/15 "2012-12-13T15:39:04Z")

</div>

David,

that was just half-ready code, without the exclusions. Yes, the app  
resolver is able to exclude elasticsearch itself, I just went to get some  
sleep 🙂

Of course, JBoss Shrinkwrap can do what Maven can, so excluding  
dependencies is not a problem.

The code - still a bit poor 🙂 - is now looking like this:

```
    String mavenSettingsFile = settings.get("apps.settings", 

```

"config/settings-es.xml");  
final MavenDependencyExclusion exclusion = MavenDependencies  
.createExclusion("org.elasticsearch:elasticsearch");  
MavenResolvedArtifact[] artifacts = null;  
String[] defaultAppClasses =  
settings.getAsArray("apps.dependencies");  
if (defaultAppClasses.length \> 0) {  
MavenDependency[] defaultDeps = new  
MavenDependency[defaultAppClasses.length];  
for (int i = 0; i \< defaultAppClasses.length; i++) {  
logger.info("adding dependency {}", defaultAppClasses[i]);  
defaultDeps[i] =  
MavenDependencies.createDependency(defaultAppClasses[i],  
ScopeType.COMPILE, false, exclusion);  
}  
artifacts = Maven.configureResolver()  
.fromFile(mavenSettingsFile)  
.addDependencies(defaultDeps)  
.resolve()  
.withMavenCentralRepo(true)  
.withTransitivity()  
.asResolvedArtifact();  
}

It's still not complete. New parameters will be provided to  
config/elasticsearch.yml to control the app resolving.

Also note, conf/settings-es.xml is a full functional Maven settings.xml,  
with all your favorite Maven-specific customized repositories, and so on.

For site plugins, there will be just URLs to ZIP files for download, no  
Maven artifacts.

I the app system, you should also be able to execute something like

bin/apps -update

and all your current plugins will be updated to the latest version!

So I am heading towards an app resolver class that is able to load all  
plugins with dependency resolution, versioning, more information via REST  
endpoints, and all the fancy stuff!

Cheers,

Jörg

--

---

<div class="post-metadata">

**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\
**Post date:** [December 16, 2012, 8:50am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/16 "2012-12-16T08:50:44Z")

</div>

Hi Ivan,

I think what Jörg suggests will still allow this.

1. An plugin author can still prepare plugin distribution including all its  
dependencies as a singlt file using shade (  
[Apache Maven Shade Plugin – Introduction](http://maven.apache.org/plugins/maven-shade-plugin/)) - btw, that is what  
elasticsearch already uses to include its own dependencies into its JAR.

2. Then the author can upload this result artefact (_plugin_.jar) anywhere  
on the internet (it can be github for example) and tell people how to add  
maven repository (or probably instruct the elasticsearch-app on command  
line?) to download and install it into ES node.

At least that is how I understand what Jörg is suggesting and started  
working on.

Yes, there might be some extra step compared to current github plugin  
distribution model but that model is going away anyway.

One very nice benefit is the fact that plugins can reuse common code and  
declare dependencies on each other (and using Maven pom for this means we  
do not have to reinvent the wheel). Going forward I do not think we can  
avoid more developed/advanced plugin dependency model anyway. It will also  
bring a lot of advantages. For example think of plugin that rely on other  
plugins. Then the only think the end user has to think about is installing  
the "top most" plugin. All the other plugins will be downloaded/updated  
behind the scene.

Regards,  
Lukas

On Wed, Dec 12, 2012 at 4:46 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:

> Part of the beauty of having the plugins as Github downloads is that the  
> barrier of entry for anyone to distribute an ES plugin was quite low.  
> Anyone that writes a plugin can distribute it easily. I am afraid that  
> every other solution would not be as easy and will hinder contributions.

--

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 17, 2012, 8:59am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/17 "2012-12-17T08:59:59Z")

</div>

Exactly, I am experimenting with an app tool that can work with three types  
of apps. "Legacy zip plugin" apps (as we know it, URL-based, without  
maven-based auto update), "site zip plugin" apps (URL-based, without Java  
classpath plugin declaration), and a new kind of "artifact apps" (a jar  
artifact from an unzipped plugin that exist in an arbitrary Maven repo).

There is an extra step and more work when releasing legacy zip plugins I  
think. Currently, plugin authors must create a special ES plugin formatted  
zip package with the maven assembly and move it manually to a location  
where it can be downloaded. But a Maven jar plugin author will be able to  
create and deploy everything by a single command "mvn deploy", which is  
less work, see  
[http://maven.apache.org/plugins/maven-deploy-plugin/usage.html](http://maven.apache.org/plugins/maven-deploy-plugin/usage.html)

I will also give a documentation how a Maven repo can be maintained on  
github, by deploying an ES app locally to a github repo, and committing it  
to github, as an extra step. Certainly not a best-practice approach, Maven  
central should be of course the preferred way.

By declaring the Maven artifacts in config/elasticsearch.yml as a list of  
artifact coordinates (group ID, artifact ID, and version), ES apps could be  
installed and maintained including dependencies and versioning.

Just a "bin/apps -update" could go through the list of apps installed,  
modify the version descriptor in the coordinate, and run a dependency  
resolution to download the latest version.

Even plugins that depend on other plugins would be easy to organize. Just  
imagine an example: One author learns about Lucene 4, writes a simple  
Lucene 4 codec with a little bit of ES integration, and publishes this as  
an ES app via a Maven repo. It is neat, but not easy to use by everybody in  
the community because it is not exposing REST endpoints, for example.  
Another author adds this ES app as a dependency to his pom.xml, completes  
the integration, writes REST endpoints for this plugin, adds more tools,  
and publishes this also as an ES app via a maven repo.

Discoverability at remote places is a harder task. Some clever repo  
browsing would be needed. Resolving artifacts in a repo works if a full  
Maven coordinate is already known from configuration. As it looks now, all  
candidate artifacts have to be downloaded into the local maven repo. Each  
artifact must be opened in a loop, and a temporary classpath must be  
examined for the es-plugin.properties resources, the ES plugin  
identification. To limit the time-consuming traversal, I think of something  
like "bin/app -discover ", so only a certain Maven group could be  
examined. This can behave like a "one stop shop".

And maybe later, in an advanced app tool version in the future, the local  
Maven repo index could automatically be indexed by ES itself to enable a  
cool app search - hello, ES maven repo river? 🙂

Cheers,

Jörg

On Sunday, December 16, 2012 9:50:44 AM UTC+1, Lukáš Vlček wrote:

> Hi Ivan,
> 
> I think what Jörg suggests will still allow this.
> 
> 1. An plugin author can still prepare plugin distribution including all  
> its dependencies as a singlt file using shade (  
> [Apache Maven Shade Plugin – Introduction](http://maven.apache.org/plugins/maven-shade-plugin/)) - btw, that is what  
> elasticsearch already uses to include its own dependencies into its JAR.
> 
> 2. Then the author can upload this result artefact (_plugin_.jar) anywhere  
> on the internet (it can be github for example) and tell people how to add  
> maven repository (or probably instruct the elasticsearch-app on command  
> line?) to download and install it into ES node.
> 
> At least that is how I understand what Jörg is suggesting and started  
> working on.
> 
> Yes, there might be some extra step compared to current github plugin  
> distribution model but that model is going away anyway.
> 
> One very nice benefit is the fact that plugins can reuse common code and  
> declare dependencies on each other (and using Maven pom for this means we  
> do not have to reinvent the wheel). Going forward I do not think we can  
> avoid more developed/advanced plugin dependency model anyway. It will also  
> bring a lot of advantages. For example think of plugin that rely on other  
> plugins. Then the only think the end user has to think about is installing  
> the "top most" plugin. All the other plugins will be downloaded/updated  
> behind the scene.
> 
> Regards,  
> Lukas
> 
> On Wed, Dec 12, 2012 at 4:46 PM, Ivan Brusic \<[iv...@brusic.com](mailto:iv...@brusic.com)\<javascript:\>
> 
> > wrote:
> 
> > Part of the beauty of having the plugins as Github downloads is that the  
> > barrier of entry for anyone to distribute an ES plugin was quite low.  
> > Anyone that writes a plugin can distribute it easily. I am afraid that  
> > every other solution would not be as easy and will hinder contributions.

--

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:59am UTC](https://discuss.elastic.co/t/manual-zip-uploads-to-github-disabled/10049/18 "2017-07-06T02:59:40Z")

</div>