I started having a weird problem where using logstash-plugin to install logstash-output-opensearch was leaving my plugin environment in a broken state.
Logstash version: logstash 7.16.1
using bundled JDK version
I started seeing this in my test environments when installing logstash plugins
root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-output-opensearch Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Validating logstash-output-opensearch Resolving mixin dependencies Updating mixin dependencies logstash-mixin-ecs_compatibility_support Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same Installing logstash-output-opensearch Killed
After this error happens, the plugin environment is left in a broken state for any logstash-plugin command, even lists or installing other plugins
---- Begin output of bin/logstash-plugin install logstash-output-lumberjack ---- STDOUT: Using bundled JDK: /usr/share/logstash/jdk Validating logstash-output-lumberjack Resolving mixin dependencies STDERR: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Bundler::ProductionError: You are trying to install in deployment mode after changing your Gemfile. Run `bundle install` elsewhere and add the updated Gemfile.lock to version control. If this is a development machine, remove the /usr/share/logstash/Gemfile freeze by running `bundle config unset frozen`. The dependencies in your gemfile changed You have added to the Gemfile: * logstash-output-opensearch ensure_equivalent_gemfile_and_lockfile at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/definition.rb:396 setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/runtime.rb:13 setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler.rb:150 setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:79 update_logstash_mixin_dependencies at /usr/share/logstash/lib/pluginmanager/install.rb:187 execute at /usr/share/logstash/lib/pluginmanager/install.rb:77 run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68 execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11 run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68 run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133 <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64 ---- End output of bin/logstash-plugin install logstash-output-lumberjack ----
When I tried as a test to just install that dependency plugin manually, I got a failure reporting that it was not a valid logstash plugin
root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-mixin-ecs_compatibility_support Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. Validating logstash-mixin-ecs_compatibility_support logstash-mixin-ecs_compatibility_support is not a Logstash plugin ERROR: Installation aborted, verification failed for logstash-mixin-ecs_compatibility_support
This appears to be because the logstash-mixin-ecs_compatibility_support gemspec doesn't have the metadata defined? Though that doesn't seem new so I'm confused why this recently started happening.
Regardless, if I install the logstash-mixin-ecs_compatibility_support plugin with the
--no-verify flag, then that installation succeeds. Then I can install the logstash-output-opensearch plugin and that also succeeds. Likewise installing the logstash-output-opensearch plugin itself with the
--no-verify flag succeeds.
I wasn't sure if this was a real bug or just something weird in our environment that I can't pinpoint, but it seems like two problems:
- logstash-plugin fails to install the logstash-mixin-ecs_compatibility_support dependency, possibly due to missing metadata?
- logstash-plugin did not gracefully handle the failure of installing a dependency plugin, leaving the plugin environment in a broken state
- This is all actually a problem with logstash-output-opensearch (which I understand is not maintained by Elastic), even though it can be remediated by manually installing the dependency with no-verify first.