# Logstash new plugin system too restrictive

**URL:** <https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247>\
**Category:** Logstash\
**Created:** [June 24, 2015, 9:29am UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247 "2015-06-24T09:29:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)\
**Post date:** [June 24, 2015, 9:29am UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/1 "2015-06-24T09:29:41Z")

</div>

With 1.5.0RC4 the --pluginpath parameter was dropped, because to paraphrase it was not needed anymore.

The new plugin system is well designed, but:

1. It breaks any installation using the pluginpath parameter. Offering no backwards compatibility is not very nice.

2. Less fun hacking on a plugin. Before the change writing of a single Ruby file and putting it in the correct directory structure was all you had to do to create a new plugin. Now you need to make a Ruby Gem, meaning you need to edit/manage some very Ruby specific files.

3. Plugins are now part of the logstash installation. If you run multiple logstash instances on a host all instances now see the plugin. Can you even install multiple versions of a plugin and use a specific one?

Please please!!!! Re-add the --pluginpath parameter:

- as a backwards compatibility to make hacking on plugins fun again
- keep the logstash installation clean and allow any instance to run with its own set of plugins.

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [June 24, 2015, 2:33pm UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/2 "2015-06-24T14:33:45Z")

</div>

Thanks for raising this @rtoma - we certainly want you to stay happy & continue hacking on plugins 😄

First, there is [this issue](https://github.com/elastic/logstash/issues/3227) which also relates to this.

Let me provide some thoughts on your points:

1- we'll be following up with some recommendations on this and see if we need to change/add anything to make this easier for you.

2- gemifying your plugin is really not that difficult, it's basically just adding a .gemspec file.

3- not sure I understand what you mean here. each logstash installation is self-contained. plugins installed in one instance are not visible to another instance. _normally_ only one version of a plugin is installed. you can install a specific version and you can update to latest version but there is no concept of having multiple versions installed and choosing a specific one - you simply install a specific version (or latest) or you update to latest.

Colin

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [June 24, 2015, 4:26pm UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/3 "2015-06-24T16:26:11Z")

</div>

As a temporary solution, until we figure a proper fix, you can do the following:

For the sake of the example, let's say you created a filter plugin named "foobar". what you can do is create the following directory structure for your plugin:

`logstash/filters/foobar.rb`

(note the "s" in filters)

if {base\_path} is the path leading to `logstash/filters/foobar.rb`, launch logstash using:

`RUBYLIB={base_path} bin/logstash ...`

and that should allow you to use filter plugin "foobar" in your config.

Let me know if this works for you.

Colin

---

<div class="post-metadata">

**Author:** ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)\
**Post date:** [June 24, 2015, 6:49pm UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/4 "2015-06-24T18:49:52Z")

</div>

Hi Colin,

Thanks so much for your quick and helpful reply.

First to answer your question about my 3rd point ("plugins are now part of the logstash installation"). I am running multiple logstash instances/jvms on a single physical many-core server. Installing a plugin with the pluginmanager will make the plugin become part of the logstash installation. All instances/jvms running on the same server can use this installed plugin. With the old --pluginpath method I could introduce a specific plugin (and even a specific plugin version, for testing/canary purposes) to a specific logstash instance/jvm. With the pluginmanager its all or nothing. That's a step back.

Your RUBYLIB= suggestion is very nice and works! Here, I will show you (gives me a chance to testdrive discuss markup 😄):

Using docker I start a logstash container configured with a small config, a silly string-reversing filter plugin on filesystem and the RUBYLIB environment var pointing to the filter plugin.

To start the container I use docker-compose (ex fig). Here is the 'docker-compose.yml' file:

```
logstash:
  image: logstash
  volumes:
    - .:/plugins
  environment:
    - RUBYLIB=/plugins
  command: |
    -e '
      input {
        heartbeat {
          interval => 5
          message => 'logstash'
        }
      }
      filter {
        reverse { }
      }
      output {
        stdout {
          codec => rubydebug
        }
      }'

```

Here is the sourcecode of the custom filter plugin, saved as logstash/filters/reverse.rb (and available in the container as /plugins/logstash/filters/reverse.rb):

```
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"

class LogStash::Filters::Reverse < LogStash::Filters::Base
  config_name "reverse"

  config :field, :validate => :string, :default => 'message'

  public
  def register
  end # def register

  public
  def filter(event)
    return unless filter?(event)
    return unless event.include?(@field)
    event[@field] = event[@field].reverse
  end # def filter

end

```

When I start the pre-configured container everything comes together and works as you suggested:

```
$ fig up
Recreating dockerlogstasheasyplugindev_logstash_1...
Attaching to dockerlogstasheasyplugindev_logstash_1
logstash_1 | {:timestamp=>"2015-06-24T18:40:11.575000+0000", :message=>"reverse plugin doesn't have a version. This plugin isn't well\n supported by the community and likely has no maintainer.", :level=>:warn}
logstash_1 | {
logstash_1 | "message" => "hsatsgol",
logstash_1 | "host" => "81f29fa6d255",
logstash_1 | "@version" => "1",
logstash_1 | "@timestamp" => "2015-06-24T18:40:11.637Z"
logstash_1 | }

```

Thanks!

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [June 25, 2015, 2:28am UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/5 "2015-06-25T02:28:27Z")

</div>

Great stuff! Happy this was helpful.

Ok, Now I understand your use-case, you actually launch 3 separate instances of logstash but from the same installation directory.

I am not convinced we should actually try to support the concept of having multiple installed plugins versions and be able to force a specific version per launched instance. This seems like a rare use-case and it's totally possible to install multiple separate instances of logstash and then install specific versions of plugins in specific instances.

Maybe take a look at the upcoming plugins packaging feature in [https://github.com/elastic/logstash/pull/3404](https://github.com/elastic/logstash/pull/3404) - this might provide an alternative for packaging plugins for specific logstash instances?

Colin

---

<div class="post-metadata">

**Author:** ![rtoma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtoma/32/44812_2.png) [@rtoma](https://discuss.elastic.co/u/rtoma)\
**Post date:** [June 25, 2015, 7:36pm UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/6 "2015-06-25T19:36:50Z")

</div>

We use the official logstash rpm.

Regarding your comment:

> [@colinsurprenant](#):
>
> [..] it's totally possible to install multiple separate instances of logstash and then install specific versions of plugins in specific instances.

How would you go about doing this? Symlinks or using the RUBYLIB trick?

.

Thanks for referencing PR 3404. That will make many engineers (and security folks) happy.

---

<div class="post-metadata">

**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\
**Post date:** [July 6, 2015, 5:31pm UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/7 "2015-07-06T17:31:40Z")

</div>

@rtoma ah yes, using the rpm. I was thinking about installing 3 instances in 3 separate directories but that would have to be with using the tarball packages. Otherwise using different `RUBYLIB` env vars per instance would be possible but not very practical for "released" plugins...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:35am UTC](https://discuss.elastic.co/t/logstash-new-plugin-system-too-restrictive/24247/8 "2017-07-06T05:35:29Z")

</div>