Keeping user installed plugins after logstash update

[hydrapolic (Tomáš Mózes)]

After installing logstash 5.4.1:
# emerge =app-admin/logstash-5.4.1

And setting up a config to include gelf as output:
output {
gelf {
host => "graylog.example.com"
}
}

Logstash fails to start because it cannot find the plugin (that's normal):

# /etc/init.d/logstash start

• Caching service dependencies ... [ ok ]
• Checking your configuration ...
  Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
  [2017-07-07T14:47:35,313][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/opt/logstash/data/queue"}
  [2017-07-07T14:47:35,603][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"output", :name=>"gelf", :path=>"logstash/outputs/gelf", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:221:in namespace_lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:157:inlegacy_lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:inlookup_pipeline_plugin'", "/opt/logstash/logstash-core/lib/logstash/plugin.rb:139:in lookup'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:91:inplugin'", "(eval):24:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:63:in initialize'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:259:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:185:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/opt/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
  [2017-07-07T14:47:35,606][FATAL][logstash.runner ] The given configuration is invalid. Reason: Couldn't find any output plugin named 'gelf'. Are you sure this is correct? Trying to load the gelf output plugin resulted in this error: Problems loading the requested plugin named gelf of type output. Error: NameError NameError
• Configuration error. Please fix your configuration files. [ !! ]
• ERROR: logstash failed to start

After installing the plugin:

# JARS_SKIP='true' bin/logstash-plugin install logstash-output-gelf
Validating logstash-output-gelf
Installing logstash-output-gelf
Installation successful

It starts properly:

# /etc/init.d/logstash start

• Checking your configuration ...
  Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
  Configuration OK
  [2017-07-07T14:50:01,343][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash [ ok ]
• /opt/logstash/data: correcting mode
• /run/logstash: creating directory
• Starting logstash ...

We update logstash to version 5.4.3:

# emerge -u app-admin/logstash-bin

And it fails to load because the plugin is gone:

# /etc/init.d/logstash restart

• Caching service dependencies ... [ ok ]
• Checking your configuration ...
  Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
  [2017-07-07T14:53:19,401][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"output", :name=>"gelf", :path=>"logstash/outputs/gelf", :error_message=>"NameError", :error_class=>NameError, :error_backtrace=>["/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:221:in namespace_lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:157:inlegacy_lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:133:in lookup'", "/opt/logstash/logstash-core/lib/logstash/plugins/registry.rb:175:inlookup_pipeline_plugin'", "/opt/logstash/logstash-core/lib/logstash/plugin.rb:139:in lookup'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:91:inplugin'", "(eval):20:in initialize'", "org/jruby/RubyKernel.java:1079:ineval'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:63:in initialize'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:259:inexecute'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:185:inrun'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/opt/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
  [2017-07-07T14:53:19,408][FATAL][logstash.runner ] The given configuration is invalid. Reason: Couldn't find any output plugin named 'gelf'. Are you sure this is correct? Trying to load the gelf output plugin resulted in this error: Problems loading the requested plugin named gelf of type output. Error: NameError NameError
• Configuration error. Please fix your configuration files. [ !! ]
• ERROR: logstash failed to start

Is there a way to retain user installed plugins during logstash updates? The problem seems to be that even though the files are in the logstash installation directory, the Gemfile is updated and the user plugins are removed from it.

If we re-install the plugin manually after the logstash update, it will work of course.

Gentoo bug: https://bugs.gentoo.org/show_bug.cgi?id=622602

Any help is appreciated.
Thanks,
Tomas