Errors installing logstash-mixin-ecs_compatibility_support plugin

I started having a weird problem where using logstash-plugin to install logstash-output-opensearch was leaving my plugin environment in a broken state.

Logstash version: logstash 7.16.1
using bundled JDK version

I started seeing this in my test environments when installing logstash plugins

root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-output-opensearch
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-output-opensearch
Resolving mixin dependencies
Updating mixin dependencies logstash-mixin-ecs_compatibility_support
Bundler attempted to update logstash-mixin-ecs_compatibility_support but its version stayed the same
Installing logstash-output-opensearch

After this error happens, the plugin environment is left in a broken state for any logstash-plugin command, even lists or installing other plugins

---- Begin output of bin/logstash-plugin install logstash-output-lumberjack ----
STDOUT: Using bundled JDK: /usr/share/logstash/jdk
Validating logstash-output-lumberjack
Resolving mixin dependencies
STDERR: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Bundler::ProductionError: You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated Gemfile.lock to version control.

If this is a development machine, remove the /usr/share/logstash/Gemfile freeze 
by running `bundle config unset frozen`.

The dependencies in your gemfile changed

You have added to the Gemfile:
* logstash-output-opensearch

  ensure_equivalent_gemfile_and_lockfile at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/definition.rb:396
                                   setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler/runtime.rb:13
                                   setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.2.33/lib/bundler.rb:150
                                  setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:79
      update_logstash_mixin_dependencies at /usr/share/logstash/lib/pluginmanager/install.rb:187
                                 execute at /usr/share/logstash/lib/pluginmanager/install.rb:77
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                                 execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
                                  <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64
---- End output of bin/logstash-plugin install logstash-output-lumberjack ----

When I tried as a test to just install that dependency plugin manually, I got a failure reporting that it was not a valid logstash plugin

root@dokken:/usr/share/logstash# bin/logstash-plugin install logstash-mixin-ecs_compatibility_support
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Validating logstash-mixin-ecs_compatibility_support
logstash-mixin-ecs_compatibility_support is not a Logstash plugin
ERROR: Installation aborted, verification failed for logstash-mixin-ecs_compatibility_support 

This appears to be because the logstash-mixin-ecs_compatibility_support gemspec doesn't have the metadata defined? Though that doesn't seem new so I'm confused why this recently started happening.

Regardless, if I install the logstash-mixin-ecs_compatibility_support plugin with the --no-verify flag, then that installation succeeds. Then I can install the logstash-output-opensearch plugin and that also succeeds. Likewise installing the logstash-output-opensearch plugin itself with the --no-verify flag succeeds.

I wasn't sure if this was a real bug or just something weird in our environment that I can't pinpoint, but it seems like two problems:

  1. logstash-plugin fails to install the logstash-mixin-ecs_compatibility_support dependency, possibly due to missing metadata?
  2. logstash-plugin did not gracefully handle the failure of installing a dependency plugin, leaving the plugin environment in a broken state

or possibly

  1. This is all actually a problem with logstash-output-opensearch (which I understand is not maintained by Elastic), even though it can be remediated by manually installing the dependency with no-verify first.

Opensearch is an AWS run product and differs from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

This open issue is related.

Hmm it does look similar but I'll note that I'm not making any manual edits to a Gemfile, I'm exclusively using the logstash-plugin command. And I don't end up in this state if I install the plugin with --no-verify

Just as a follow-up here, I want to add that have had some testing with another co-worker, and they are not experiencing the failures I am, even though we're running the same Dockerfile and chef recipes, so that does support the premise that there is just something off in my local environment that is leading to some abnormal failures. It's still unclear why this is happening but I don't have enough reproducible data to fairly claim it is an issue with elastic code.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.