Logstash new plugin system too restrictive

Elastic Stack Logstash
1

With 1.5.0RC4 the --pluginpath parameter was dropped, because to paraphrase it was not needed anymore.

The new plugin system is well designed, but:

  1. It breaks any installation using the pluginpath parameter. Offering no backwards compatibility is not very nice.

  2. Less fun hacking on a plugin. Before the change writing of a single Ruby file and putting it in the correct directory structure was all you had to do to create a new plugin. Now you need to make a Ruby Gem, meaning you need to edit/manage some very Ruby specific files.

  3. Plugins are now part of the logstash installation. If you run multiple logstash instances on a host all instances now see the plugin. Can you even install multiple versions of a plugin and use a specific one?

Please please!!!! Re-add the --pluginpath parameter:

  • as a backwards compatibility to make hacking on plugins fun again
  • keep the logstash installation clean and allow any instance to run with its own set of plugins.
2

Thanks for raising this @rtoma - we certainly want you to stay happy & continue hacking on plugins :smile:

First, there is this issue which also relates to this.

Let me provide some thoughts on your points:

1- we'll be following up with some recommendations on this and see if we need to change/add anything to make this easier for you.

2- gemifying your plugin is really not that difficult, it's basically just adding a .gemspec file.

3- not sure I understand what you mean here. each logstash installation is self-contained. plugins installed in one instance are not visible to another instance. normally only one version of a plugin is installed. you can install a specific version and you can update to latest version but there is no concept of having multiple versions installed and choosing a specific one - you simply install a specific version (or latest) or you update to latest.

Colin

3

As a temporary solution, until we figure a proper fix, you can do the following:

For the sake of the example, let's say you created a filter plugin named "foobar". what you can do is create the following directory structure for your plugin:

logstash/filters/foobar.rb

(note the "s" in filters)

if {base_path} is the path leading to logstash/filters/foobar.rb, launch logstash using:

RUBYLIB={base_path} bin/logstash ...

and that should allow you to use filter plugin "foobar" in your config.

Let me know if this works for you.

Colin

4

Hi Colin,

Thanks so much for your quick and helpful reply.

First to answer your question about my 3rd point ("plugins are now part of the logstash installation"). I am running multiple logstash instances/jvms on a single physical many-core server. Installing a plugin with the pluginmanager will make the plugin become part of the logstash installation. All instances/jvms running on the same server can use this installed plugin. With the old --pluginpath method I could introduce a specific plugin (and even a specific plugin version, for testing/canary purposes) to a specific logstash instance/jvm. With the pluginmanager its all or nothing. That's a step back.

Your RUBYLIB= suggestion is very nice and works! Here, I will show you (gives me a chance to testdrive discuss markup :smile:):

Using docker I start a logstash container configured with a small config, a silly string-reversing filter plugin on filesystem and the RUBYLIB environment var pointing to the filter plugin.

To start the container I use docker-compose (ex fig). Here is the 'docker-compose.yml' file:

logstash:
  image: logstash
  volumes:
    - .:/plugins
  environment:
    - RUBYLIB=/plugins
  command: |
    -e '
      input {
        heartbeat {
          interval => 5
          message => 'logstash'
        }
      }
      filter {
        reverse { }
      }
      output {
        stdout {
          codec => rubydebug
        }
      }'

Here is the sourcecode of the custom filter plugin, saved as logstash/filters/reverse.rb (and available in the container as /plugins/logstash/filters/reverse.rb):

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"

class LogStash::Filters::Reverse < LogStash::Filters::Base
  config_name "reverse"

  config :field, :validate => :string, :default => 'message'

  public
  def register
  end # def register

  public
  def filter(event)
    return unless filter?(event)
    return unless event.include?(@field)
    event[@field] = event[@field].reverse
  end # def filter

end

When I start the pre-configured container everything comes together and works as you suggested:

$ fig up
Recreating dockerlogstasheasyplugindev_logstash_1...
Attaching to dockerlogstasheasyplugindev_logstash_1
logstash_1 | {:timestamp=>"2015-06-24T18:40:11.575000+0000", :message=>"reverse plugin doesn't have a version. This plugin isn't well\n supported by the community and likely has no maintainer.", :level=>:warn}
logstash_1 | {
logstash_1 |        "message" => "hsatsgol",
logstash_1 |           "host" => "81f29fa6d255",
logstash_1 |       "@version" => "1",
logstash_1 |     "@timestamp" => "2015-06-24T18:40:11.637Z"
logstash_1 | }

Thanks!

5

Great stuff! Happy this was helpful.

Ok, Now I understand your use-case, you actually launch 3 separate instances of logstash but from the same installation directory.

I am not convinced we should actually try to support the concept of having multiple installed plugins versions and be able to force a specific version per launched instance. This seems like a rare use-case and it's totally possible to install multiple separate instances of logstash and then install specific versions of plugins in specific instances.

Maybe take a look at the upcoming plugins packaging feature in https://github.com/elastic/logstash/pull/3404 - this might provide an alternative for packaging plugins for specific logstash instances?

Colin

6

We use the official logstash rpm.

Regarding your comment:

How would you go about doing this? Symlinks or using the RUBYLIB trick?

.

Thanks for referencing PR 3404. That will make many engineers (and security folks) happy.

7

@rtoma ah yes, using the rpm. I was thinking about installing 3 instances in 3 separate directories but that would have to be with using the tarball packages. Otherwise using different RUBYLIB env vars per instance would be possible but not very practical for "released" plugins...