I started having a weird problem where using logstash-plugin to install logstash-output-opensearch was leaving my plugin environment in a broken state.
Logstash version: logstash 7.16.1
using bundled JDK version
I started seeing this in my test environments when installing logstash plugins
root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-output-opensearch
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-opensearch
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Installing logstash-output-opensearch
Killed
After this error happens, the plugin environment is left in a broken state for any logstash-plugin command, even lists or installing other plugins
---- Begin output of bin/logstash-plugin install logstash-output-lumberjack ----
STDOUT: Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-output-lumberjack
Resolving mixin dependencies
STDERR: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Bundler::ProductionError: You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated Gemfile.lock to version control.
If this is a development machine, remove the /usr/share/logstash/Gemfile freeze
by running `bundle config unset frozen`.
The dependencies in your gemfile changed
You have added to the Gemfile:
* logstash-output-opensearch
ensure_equivalent_gemfile_and_lockfile at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/definition.rb:396
setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/runtime.rb:13
setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler.rb:150
setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:79
update_logstash_mixin_dependencies at /usr/share/logstash/lib/pluginmanager/install.rb:187
execute at /usr/share/logstash/lib/pluginmanager/install.rb:77
run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
<main> at /usr/share/logstash/lib/pluginmanager/main.rb:64
---- End output of bin/logstash-plugin install logstash-output-lumberjack ----
When I tried as a test to just install that dependency plugin manually, I got a failure reporting that it was not a valid logstash plugin
root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-mixin-ecs_compatibility_support
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-mixin-ecs_compatibility_support
logstash-mixin-ecs_compatibility_support is not a Logstash plugin
ERROR: Installation aborted, verification failed for logstash-mixin-ecs_compatibility_support
This appears to be because the logstash-mixin-ecs_compatibility_support gemspec doesn't have the metadata defined? Though that doesn't seem new so I'm confused why this recently started happening.
Regardless, if I install the logstash-mixin-ecs_compatibility_support plugin with the --no-verify flag, then that installation succeeds. Then I can install the logstash-output-opensearch plugin and that also succeeds. Likewise installing the logstash-output-opensearch plugin itself with the --no-verify flag succeeds.
I wasn't sure if this was a real bug or just something weird in our environment that I can't pinpoint, but it seems like two problems:
- logstash-plugin fails to install the logstash-mixin-ecs_compatibility_support dependency, possibly due to missing metadata?
- logstash-plugin did not gracefully handle the failure of installing a dependency plugin, leaving the plugin environment in a broken state
or possibly
- This is all actually a problem with logstash-output-opensearch (which I understand is not maintained by Elastic), even though it can be remediated by manually installing the dependency with no-verify first.
)