Following documentation for editing the Gemfile breaks Logstash

Elastic Stack Logstash
1

I need to make Logstash 7 on a server that doesn't have Internet access use a local Gem repository. If I follow the instructions at Private Gem Repositories | Logstash Reference [7.17] | Elastic
then instead of

After saving the new version of the gemfile, use plugin management commands normally.

I find that Logstash no longer works.

root@fedora:~# rpm -qa | grep logstash
logstash-7.17.27-1.x86_64
root@fedora:~# head -5 /usr/share/logstash/Gemfile
# This is a Logstash generated Gemfile.
# If you modify this file manually all comments and formatting will be lost.

source "https://rubygems.org"
gem "logstash-core", :path => "./logstash-core"
root@fedora:~# /usr/share/logstash/bin/logstash-plugin list | grep codec 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash-codec-avro
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
root@fedora:~# sed -i 's/rubygems.org/foo.bar/' /usr/share/logstash/Gemfile
root@fedora:~# head -5 /usr/share/logstash/Gemfile
# This is a Logstash generated Gemfile.
# If you modify this file manually all comments and formatting will be lost.

source "https://foo.bar"
gem "logstash-core", :path => "./logstash-core"
root@fedora:~# /usr/share/logstash/bin/logstash-plugin list | grep codec 
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Bundler::ProductionError: You are trying to install in deployment mode after changing
your Gemfile. Run `bundle install` elsewhere and add the
updated ../usr/share/logstash/Gemfile.lock to version control.

If this is a development machine, remove the /usr/share/logstash/Gemfile freeze 
by running `bundle config unset frozen`.

The list of sources changed

  ensure_equivalent_gemfile_and_lockfile at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:410
                                   setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/runtime.rb:13
                                   setup at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:162
                                  setup! at /usr/share/logstash/lib/bootstrap/bundler.rb:87
                                 execute at /usr/share/logstash/lib/pluginmanager/list.rb:33
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                                 execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/subcommand/execution.rb:11
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68
                                     run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133
                                  <main> at /usr/share/logstash/lib/pluginmanager/main.rb:64
root@fedora:~#

Also if I try to then start logstash.service it fails with

Feb 10 14:43:26 fedora systemd[1]: Started logstash.service - logstash.
Feb 10 14:43:26 fedora logstash[2852]: Using bundled JDK: /usr/share/logstash/jdk
Feb 10 14:43:26 fedora logstash[2852]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 10 14:43:30 fedora logstash[2852]: [FATAL] 2025-02-10 14:43:30.868 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
Feb 10 14:43:30 fedora logstash[2852]: If you are a user, this is a bug.
Feb 10 14:43:30 fedora logstash[2852]: If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.
Feb 10 14:43:30 fedora systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 14:43:30 fedora systemd[1]: logstash.service: Failed with result 'exit-code'.

This is not a "development machine" but I tried figuring out what is meant by running bundle… anyway

root@fedora:~# which bundle
/usr/bin/which: no bundle in (/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/var/lib/snapd/snap/bin)
root@fedora:~# find /usr/share/logstash/ -name bundle
/usr/share/logstash/vendor/bundle
/usr/share/logstash/vendor/bundle/jruby/2.5.0/bin/bundle
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/exe/bundle
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rake-12.3.3/bin/bundle
/usr/share/logstash/vendor/jruby/lib/ruby/gems/shared/gems/rake-12.3.3/bin/bundle
root@fedora:~# /usr/share/logstash/vendor/bundle/jruby/2.5.0/bin/bundle config unset frozen
-bash: /usr/share/logstash/vendor/bundle/jruby/2.5.0/bin/bundle: cannot execute: required file not found
root@fedora:~#

I've found some references to this problem being due to Logstash now freezing a bundle or something, I don't understand it and haven't found a solution. But I'm thinking I shouldn't need to be looking for a solution, because I am doing something the documentation for Logstash 7.17 says I can do with Logstash 7.17. That Logstash breaks after I do it seems to mean one of two possibilities, that I am not following the documentation correctly (and I can't see how anything I'm doing wrong), or that the documentation is wrong (but the discussions I find about freezing bundles from 2021, which would mean the documentation has been wrong for over three years.)

Can someone point out what I'm doing differently to what the documentation says to do, or explain what I need to do as well as or instead of what the documentations says?

Edit: I forgot to say I tried

 If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set

even though I am not a developer, it doesn't help.

root@fedora:~# /usr/share/logstash/bin/logstash-plugin list --enable-local-plugin-development  | grep codec
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ERROR: Unrecognised option '--enable-local-plugin-development'

See: 'bin/logstash-plugin list --help'
root@fedora:~#

and if I try editing service definition

root@fedora:~# grep Start /etc/systemd/system/logstash.service 
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" --enable-local-plugin-development
root@fedora:~#

(I know I should use as dropin if I was doing for properly) starting logstash.service results in

Feb 10 15:19:59 fedora systemd[1]: Started logstash.service - logstash.
Feb 10 15:19:59 fedora logstash[4117]: Using bundled JDK: /usr/share/logstash/jdk
Feb 10 15:19:59 fedora logstash[4117]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Feb 10 15:20:06 fedora logstash[4117]: Resolving dependencies.............................
Feb 10 15:20:07 fedora logstash[4117]: [FATAL] 2025-02-10 15:20:07.015 [main] Logstash - Logstash stopped processing because of an error: (PermissionError) Bundler::PermissionError
Feb 10 15:20:07 fedora logstash[4117]: org.jruby.exceptions.StandardError: (PermissionError) Bundler::PermissionError
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bundler_minus_2_dot_3_dot_18.lib.bundler.shared_helpers.filesystem_access(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/shared_helpers.rb:105) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bundler_minus_2_dot_3_dot_18.lib.bundler.definition.lock(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/definition.rb:323) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bundler_minus_2_dot_3_dot_18.lib.bundler.runtime.lock(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/runtime.rb:98) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bundler_minus_2_dot_3_dot_18.lib.bundler.runtime.setup(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler/runtime.rb:35) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.bundler_minus_2_dot_3_dot_18.lib.bundler.setup(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/bundler-2.3.18/lib/bundler.rb:162) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.lib.bootstrap.bundler.setup!(/usr/share/logstash/lib/bootstrap/bundler.rb:87) ~[?:?]
Feb 10 15:20:07 fedora logstash[4117]:         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
Feb 10 15:20:07 fedora systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Feb 10 15:20:07 fedora systemd[1]: logstash.service: Failed with result 'exit-code'.
2

So from looking around I eventually stumbled upon FATAL error after updating to 7.16.1 - #12 by maltewhiite and it turns out that is how to make Logstash work again after editing the Gemfile. I do this after editing the Gemfile:

$ cd /usr/share/logstash/
$ bin/ruby vendor/bundle/jruby/2.5.0/bin/bundle lock

That makes Logstash work again. I would still like to understand the discrepancy between the documentation which says edit the Gemfile and then carry on as normal and my experience that editing the Gemfile totally breaks Logstash, so if anyone is able to explain that, that would be good.

I also found that after the Gemfile was modified and Logstash wasn't just totally broken, trying to install a plugin failed (with an error I've already lost, sorry) and that to make plugin installation work it's necessary to precede the install command with a definition of LOGSTASH_PACK_URL e.g.

$ LOGSTASH_PACK_URL=http://localhost:1234 /usr/share/logstash/bin/logstash-plugin install pluginnamehere

(Nothing is listening on localhost:1234 it seems a value just has to be set.) I

Found at Airgapped install of plugin fails trying to access artifacts.elastic.co - #5 by SebastianThorn

3

I did spend tons of time to get this working.
The plugin we use have made a new release some time ago op we need to eventually rebuild the image.

I can try to rebuild is with all new releases tomorrow if i get the time.

Br Sebastian Thörn

4

I'v just built a new version based on 8.17.2 and it works.
This is a the part in our Dockerfile that is relevant

FROM base AS artifacts

# Set up envitornment-variables
ENV SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
ENV LOGSTASH_PACK_URL=http://localhost:1234
ENV LS_JAVA_OPTS="-Xms2048m -Xmx2048m"

COPY ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

RUN sed -i 's/source .*/source "https:\/\/artifactory.example.com:8443\/artifactory\/rubygems\/"/' Gemfile

# Copy and unpack logstash-plugin
WORKDIR /plugin
COPY artifacts.tar.gz /tmp/artifacts.tar.gz
RUN tar xvf /tmp/artifacts.tar.gz -C /tmp/

RUN logstash-plugin install /tmp/dist/logstash-output-opensearch-{{ opensearch_output_plugin_release }}-java.gem
RUN logstash-plugin prepare-offline-pack logstash-output-opensearch

Best Regards
Sebastian Thörn

5

Interesting that you have nothing there to deal with Logstash breaking after you edit the Gemfile, but you are using a different major version of Logstash to me, so maybe Logstash 8 doesn't totally break when the Gemfile is edited.