[FATAL] 2021-12-15 11:35:53.567 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
What is the Gemfile change?
How do I debug this?
Anyone else experienced this?
I can't find anything relevant on Google.
- how did you acquire Logstash (tarball, package registry, etc.)?
- how are you attempting to start it (command line, service manager, etc)?
- do you have any custom plugins, beyond what comes installed?
- what platform are you running on?
Logstash uses an embedded tool called Bundler to manage its plugins and their dependencies. The Gemfile contains instructions about logstash's Ruby dependencies (such as plugins), and the Gemfile.lock contains information about the dependency graph as currently installed, including dependencies-of-dependencies. Logstash now runs with the Gemfile frozen, which prevents the two from falling out of sync with each other.
These can fall out of sync if you modify the Gemfile, or if you install a plugin that is a file path on the local filesystem and go on to change that local plugin. A new flag --enable-local-plugin-development allows this validation check to be bypassed when developing local plugins.
For more info, see https://github.com/elastic/logstash/pull/13015
1 Like
We use Puppet elastic/logstash · Module for managing and configuring Logstash · Puppet Forge
class { 'logstash':
version => $version,
config_dir => $config_dir,
manage_repo => false,
settings => {
'xpack.management.enabled' => false, # Do not use centralised pipeline management
},
jvm_options => [
"-Xms${jvm_memory}",
"-Xmx${jvm_memory}",
#CVE-2021-44228 Mitigation
"-Dlog4j2.formatMsgNoLookups=true",
#XX flags are set to enable use of java11
'8-9:-XX:+UseParNewGC',
'8-9:-XX:+UseConcMarkSweepGC',
'8-9:-XX:CMSInitiatingOccupancyFraction=75',
'8-9:-XX:+UseCMSInitiatingOccupancyOnly',
],
startup_options => {
#needed https://github.com/elastic/logstash/issues/6902
'JAVACMD' => '/usr/share/logstash/jdk/bin/java',
},
pipelines => $_default_pipelines + $_distributor_pipelines,
}
# Use Artifactory for logstash plugin installation
file_line { 'gem-repo':
ensure => present,
path => '/usr/share/logstash/Gemfile',
line => "source \"${gems_repo_url}\"",
match => '^source "https://',
append_on_no_match => false,
require => Class['logstash'],
}
Thanks a lot for explaining the problem Yaauie.
So now that you've told us what the problem is, how do we fix it?
Assume I know nothing. Someone else wrote this code who is no longer with the team, and I am only looking into this, because of the Log4J vulnerability.
I tried yum remove logstash and reinstalling by puppet agent -t, but still same error.
dec 16 11:48:18 logstash[8145]: [FATAL] 2021-12-16 11:48:18.130 [main] Logstash - Logstash was unable to start due to an unexpected Gemfile change.
dec 16 11:48:18 logstash[8145]: If you are a user, this is a bug.
dec 16 11:48:18 logstash[8145]: If you are a logstash developer, please try restarting logstash with the `--enable-local-plugin-development` flag set.
How can there be "Unexpected Gemfile change", if it is a fresh install?
It is "working" if I run with flag --enable-local-plugin-development, but that is a development flag.
How do I fix this for Production??
I am not familiar with the linked Puppet recipe, but the references to Gemfile and mentions of external manipulation of plugin management is likely relevant. Was there more to the puppet config that got cropped?
Does the recipe include any logstash::plugin directives, and if so what form do they take?
From MarcoMartins86 GitHub:
I've also encountered this problem. I'm using the logstash official docker and I was adding gems to the Gemfile manually which triggers the error. The tricky part was to run the embedded bundle to install the gems, not sure if there is an easier way to do it without having hardcoded paths that might change in the future. This solved for me
/usr/share/logstash/bin/ruby -S $(/usr/share/logstash/bin/ruby -S gem env gemdir 2>&1 | grep /usr/share/logstash/vendor/bundle/)/bin/bundle add redis , if you're not adding gems manually but still see the error maybe try to do a bundle lock before running logstash.
This looks like the only usage of logstash::plugin in the code:
If I ctrl+click the logstash::plugin I only see that one usage. We don't use it in our own code.
We used to have logstash-filter-alter, but we removed it, to try to fix the issue.
On of my colleagues said that this might be the cause.
In this code we edit a line in the Gemfile.
Would that cause a Gemfile.lock issue?
And if so, how do we fix that, and still change the Gemfile line?
Firing off this command works in my Vagrant environment.
I now try writing some Puppet code that fires off this command, if we change gemfile, and then restart logstash.
/usr/share/logstash/bin/ruby /usr/share/logstash/vendor/bundle/jruby/2.5.0/bin/bundle lock
This Puppet code seems to work. It works in my Vagrant environment.
# Use Artifactory for logstash plugin installation
file_line { 'gem-repo':
ensure => present,
path => '/usr/share/logstash/Gemfile',
line => "source \"${gems_repo_url}\"",
match => '^source "https://',
append_on_no_match => false,
require => Class['logstash'],
notify => Exec['bundle-lock'],
}
exec { 'bundle-lock':
command => '/usr/share/logstash/bin/ruby -S $(/usr/share/logstash/bin/ruby -S gem env gemdir 2>&1 | grep /usr/share/logstash/vendor/bundle/)/bin/bundle lock',
refreshonly => true,
notify => Service['logstash'],
}
Moving to testing in Dev now.
In Dev I got errors.
I did:
- cd /usr/share/logstash
- bin/ruby vendor/bundle/jruby/2.5.0/bin/bundle install
- bin/ruby vendor/bundle/jruby/2.5.0/bin/bundle lock
Now I get new error:
"Your settings are invalid. Reason: Path "/var/lib/logstash/queue" must be a writable directory. It is not writable."
"Queue must be writable" makes sense. But writable for who? And how do I fix it?
This is /var/lib/logstash
changed them all to logstash:logstash via command chown and it seems to be working in Dev now
Will implement in prod on monday.
1 Like
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.