# Add plugins from classpath in embedded Elasticsearch client node

**URL:** <https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269>\
**Category:** Elasticsearch\
**Created:** [December 3, 2015, 10:10am UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269 "2015-12-03T10:10:21Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![joschi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joschi/32/44946_2.png) [@joschi](https://discuss.elastic.co/u/joschi)\
**Post date:** [December 3, 2015, 10:10am UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/1 "2015-12-03T10:10:21Z")

</div>

Hi,

I'm currently working on upgrading an application to Elasticsearch 2.1.0 and ran into some problems regarding loading Elasticsearch plugins from the classpath (instead of loading them from the `plugins` directory).

In Elasticsearch 1.x and 2.0.x it has been possible to load Elasticsearch plugins from the classpath into an embedded `Node`, but this feature has been removed in Elasticsearch 2.1.0 (see [#13055](https://github.com/elastic/elasticsearch/pull/13055)) and it's now only possible to do this for `TransportClient` via the `TransportClient.Builder#addPlugin()` method.

Someone filed an issue about it in [#13212](https://github.com/elastic/elasticsearch/issues/13212) and I created a pull request to (re-) introduce a similar method to `NodeBuilder` but it was closed (see [#15107](https://github.com/elastic/elasticsearch/pull/15107)).

I'm wondering now how I could add a classpath plugin to an embedded Elasticsearch node. For the time being I "cheated" a little bit by creating a sub-class of `Node` which exposes the constructor which allows passing a collection of plugin classes to be loaded on startup, but this feels kind of dirty.

The reason I can't simply use a `TransportClient` in my application is, that I need realtime information about the cluster state, e. g. when nodes were added/removed, or indices were created/closed/deleted, for which I've created a `ClusterStateListener` handling those Elasticsearch events and propagating them via an internal [`EventBus`](https://google.github.io/guava/releases/18.0/api/docs/com/google/common/eventbus/EventBus.html) to my application. As far as I see, using a `TransportClient` I would need to actively poll for that information in a certain interval, which would of course also increase load on the Elasticsearch cluster. Additionally, using a client node _should_ be more efficient when indexing large amounts of documents because it knows about the cluster topology (which could be done with the `TransportClient` too, when "sniff" mode is activated – but also only with a delay).

The question is: Is there another way to load plugins into an embedded Elasticsearch node from the classpath except for the "dirty" way I described above?

Cheers,  
joschi

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 3, 2015, 10:24am UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/2 "2015-12-03T10:24:11Z")

</div>

I'm unsure if this would help, but here is what I'm doing in tests.

Create a MockNode class:

```java
public class MockNode extends Node {

    // these are kept here so a copy of this MockNode can be created, since Node does not store them
    private Version version;
    private Collection<Class<? extends Plugin>> plugins;

    public MockNode(Settings settings, Version version, Collection<Class<? extends Plugin>> classpathPlugins) {
        super(settings, version, classpathPlugins);
        this.version = version;
        this.plugins = classpathPlugins;
    }

    public Collection<Class<? extends Plugin>> getPlugins() {
        return plugins;
    }

    public Version getVersion() {
        return version;
    }
}

```

Then starts a Node this way:

```java
    public static void main(String[] args) throws Throwable {
        Settings.Builder settings = Settings.builder().build();
        final CountDownLatch latch = new CountDownLatch(1);
        final Node node = new MockNode(settings.build(), Version.CURRENT, Collections.singletonList(MyClassPlugin.class));
        Runtime.getRuntime().addShutdownHook(new Thread() {
            @Override
            public void run() {
                node.close();
                latch.countDown();
            }
        });
        node.start();
        latch.await();
    }

```

Does this help?

---

<div class="post-metadata">

**Author:** ![joschi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joschi/32/44946_2.png) [@joschi](https://discuss.elastic.co/u/joschi)\
**Post date:** [December 3, 2015, 10:39am UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/3 "2015-12-03T10:39:14Z")

</div>

Thanks for your reply!

This is basically what I'm doing right now:

> For the time being I "cheated" a little bit by creating a sub-class of `Node` which exposes the constructor which allows passing a collection of plugin classes to be loaded on startup, but this feels kind of dirty.

The question is, if this is the intended way to do it.

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [December 3, 2015, 12:42pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/4 "2015-12-03T12:42:55Z")

</div>

Continuing the discussion from [https://github.com/elastic/elasticsearch/pull/15107](https://github.com/elastic/elasticsearch/pull/15107)

I agree `NodeBuilder` and embedding plugins by classpath is a very handy feature, not just because it was one of the reasons why I find ES plugin writing so attractive back in 2010.

If plugins are really no longer allowed on the classpath, the ES team should remove either the shipwrecked `NodeBuilder` class completely from the public API, which lacks plugins, or even more clearly, the ES team should send a clear signal and inform the community plugin authors to stop creating node-embedded plugins at all. By just making cumbersome API changes without clear reasoning [https://github.com/elastic/elasticsearch/issues/15060](https://github.com/elastic/elasticsearch/issues/15060) this can quickly become very uncomfortable and hard to understand.

I hope 2.2 will bring some relief regarding `NodeBuilder` and classpath loading but it's not released yet.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [December 5, 2015, 12:05pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/5 "2015-12-05T12:05:29Z")

</div>

Hi Jörg

There are two aspects to consider here. The first is security. We have made huge efforts to lock Elasticsearch down to reduce the chances that malicious code can exploit the Elasticsearch server. This lockdown is enforced via the plugin mechanism which adds:

- JarHell check
- Class loader isolation
- Privileges limited to the bare minimum by the Java security manager

Obviously your own code is not malicious but, if we add the ability for users to just bypass these protections in production, then they will. Making security opt-in defeats the object of the exercise. Getting security right is hard (just look at how many people suffer from JarHell) so if we offer an easy way out then users will choose the easy option and just bypass security.

The second aspect is testing code in the way that it will be used. Elasticsearch used to allow a million different configurations, most of which were untested and many of which had subtle edge cases. We are trying to reduce these options to a limited set which are backed by real tests and which are maintainable and supportable.

In the same way, your plugins should be tested in the way they will be used in the wild, hence using the test framework instead of a quick shortcut option that bypasses the usual checks. This means installing your plugins in the `plugins` directory and providing a `plugin-descriptor.properties` file. Client nodes already require a proper `config` directory as they are real nodes, not just transport clients. Similarly, if a client node requires plugins, those should go into the `plugins` directory.

We are doing exactly the same thing (and for the same reasons) with core parts of Elasticsearch, such as the scripting engines and the new node-ingest feature [https://github.com/elastic/elasticsearch/issues/14049](https://github.com/elastic/elasticsearch/issues/14049). We will be shipping these features as "modules" (see [https://github.com/elastic/elasticsearch/pull/15233](https://github.com/elastic/elasticsearch/pull/15233)) but in essence they are just plugins which are shipped by default.

I agree that these changes have made plugin development more cumbersome, but I think that the security benefits to our users outweigh the cost.

---

<div class="post-metadata">

**Author:** ![micpalmia](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/micpalmia/32/541_2.png) [@micpalmia](https://discuss.elastic.co/u/micpalmia)\
**Post date:** [January 6, 2016, 12:45pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/6 "2016-01-06T12:45:29Z")

</div>

Hi Clinton, thanks for the very clear answer. My use case seems somewhat simpler -and more common- than the use cases mentioned before, but I seem not to be able to find any documentation or help on this.

In production, we install our plugins with the `bin/plugin` script as we have always done, and the applications connect either via HTTP or using the transport client. With ES1.x we had also set up a few tests in our Java code base that would perform some indexing and search operations over test data, _using the production settings and mappings_. As we use a custom token filter in production, I obviously need the same token filter to be available to the embedded instance that is started while testing.

With ES1, I only needed to add the plugin JAR to the classpath. Is there no simple way to do this with ES2? How can I provide the plugin to the embedded node that, at the moment, is started like this?

```java
        Settings.Builder elasticsearchSettings = Settings.settingsBuilder()
				.put("http.enabled", "false")
				.put("path.home", homeDirectory);

		node = nodeBuilder()
				.local(true)
				.settings(elasticsearchSettings.build())
				.node();

```

Should I somehow provide a plugin directory containing (unpacked? how?) plugins to the embedded node?

---

<div class="post-metadata">

**Author:** ![marcoscarceles](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marcoscarceles/32/6339_2.png) [@marcoscarceles](https://discuss.elastic.co/u/marcoscarceles)\
**Post date:** [February 17, 2016, 1:08pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/7 "2016-02-17T13:08:39Z")

</div>

Same issue here. Trying to load a local Node to run several automated tests used to work on previous versions by including the plugin jars in the classpath. How are we meant to load these on the configured plugins directory now?

---

<div class="post-metadata">

**Author:** ![roytmana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/roytmana/32/44855_2.png) [@roytmana](https://discuss.elastic.co/u/roytmana)\
**Post date:** [February 17, 2016, 10:15pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/8 "2016-02-17T22:15:02Z")

</div>

Could we retain all the benefits of current approach but support pluggable discovery mechanism maybe based on say URLs for loading plugins (ex: if it is in WAR file, modules and plugins virtual directories could be navigated via URLs) or JAR per plugin with plugin-descriptor.properties even though it won't be loaded by a separate class loader and security probably won't apply in this case but the rest of benefits will...

---

<div class="post-metadata">

**Author:** ![pnowy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pnowy/32/19868_2.png) [@pnowy](https://discuss.elastic.co/u/pnowy)\
**Post date:** [March 1, 2016, 8:12am UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/9 "2016-03-01T08:12:15Z")

</div>

Hi,

I have the same question: is there a possibility to use the plugin on classpath for embedded node?

What about some plugin registering mechanism during embedded node configuration? Something like that:

```
Node node = new NodeBuilder().settings(settings).clusterName(clusterName).node();

```

You could add the support of the property like: 'plugin.enableClasspathScanning' or something like that.

Thanks,  
Przemyslaw

---

<div class="post-metadata">

**Author:** ![Thomas\_Beauvais](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_beauvais/32/11784_2.png) [@Thomas\_Beauvais](https://discuss.elastic.co/u/Thomas_Beauvais)\
**Post date:** [September 7, 2016, 1:17pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/10 "2016-09-07T13:17:42Z")

</div>

What is the status of this? It seems this discussion has trailed off and I am find little information around the web.

I have a very similar use case where we need to enabled plugins in an isolated Node setting testing.

---

<div class="post-metadata">

**Author:** ![pnowy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pnowy/32/19868_2.png) [@pnowy](https://discuss.elastic.co/u/pnowy)\
**Post date:** [September 7, 2016, 1:38pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/11 "2016-09-07T13:38:23Z")

</div>

Hi,

I fix this problem by extending the Node class:

```
private class NodeWithPlugins extends Node {
    protected NodeWithPlugins(Environment environment, Version version, Collection<Class<? extends Plugin>> classpathPlugins) {
        super(environment, version, classpathPlugins);
    }
}

```

and usage:

```
Node node = new NodeWithPlugins(InternalSettingsPreparer.prepareEnvironment(settings, null), Version.CURRENT,
                                    singletonList(AnalysisStempelPlugin.class)).start();

```

You could also consider something I found lately on Allegro blog:

> **[allegro/embedded-elasticsearch](https://github.com/allegro/embedded-elasticsearch)**
>
> embedded-elasticsearch - Tool that ease up creation of integration tests with Elasticsearch

Hope it helps!

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [September 7, 2016, 9:20pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/12 "2016-09-07T21:20:27Z")

</div>

Elasticsearch has made it clear that they will not support embedded  
environments anymore. This stance was highlighted in a recent blog post:

> **[Elasticsearch, the server](https://www.elastic.co/blog/elasticsearch-the-server)**
>
> How Elasticsearch is evolving into a secure, stable, reliable, predictable solution.

Ivan

---

<div class="post-metadata">

**Author:** ![billnbell](https://avatars.discourse-cdn.com/v4/letter/b/eb8c5e/32.png) [@billnbell](https://discuss.elastic.co/u/billnbell)\
**Post date:** [March 18, 2017, 8:50pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/13 "2017-03-18T20:50:24Z")

</div>

Remember you can also use:

List plugins = new ArrayList();  
plugins.add(AnalysisStempelPlugin.class);  
// add as many as you like

Node node = new NodeWithPlugins(InternalSettingsPreparer.prepareEnvironment(settings, null), Version.CURRENT, plugins).start();

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:02pm UTC](https://discuss.elastic.co/t/add-plugins-from-classpath-in-embedded-elasticsearch-client-node/36269/14 "2017-07-05T22:02:07Z")

</div>