Logstash fails to start: "no such file to load -- concurrent"

Elastic Stack Logstash
1

I recently upgraded my Elastic stack from 7.12 to 7.15.1 on Ubuntu 18.04. Logstash is running via systemd. Now Logstash fails to start with the following error:

Nov 02 19:58:28 server logstash[30340]: Using bundled JDK: /usr/share/logstash/jdk
Nov 02 19:58:36 server logstash[30340]: [FATAL] 2021-11-02 19:58:36.526 [main] Logstash - Logstash stopped processing because of an error: (LoadError) no such file to load -- concurrent
Nov 02 19:58:36 server logstash[30340]: org.jruby.exceptions.LoadError: (LoadError) no such file to load -- concurrent
Nov 02 19:58:36 server logstash[30340]:         at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.19.0.jar:?]
Nov 02 19:58:36 server logstash[30340]:         at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/config/defaults.rb:18) ~[?:?]
Nov 02 19:58:36 server logstash[30340]:         at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.19.0.jar:?]
Nov 02 19:58:36 server logstash[30340]:         at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/config/cpu_core_strategy.rb:18) ~[?:?]
Nov 02 19:58:36 server logstash[30340]:         at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.19.0.jar:?]
Nov 02 19:58:36 server logstash[30340]:         at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/environment.rb:19) ~[?:?]
Nov 02 19:58:36 server logstash[30340]:         at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.19.0.jar:?]
Nov 02 19:58:36 server logstash[30340]:         at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:38) ~[?:?]
Nov 02 19:58:36 server logstash[30340]:         at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.19.0.jar:?]
Nov 02 19:58:36 server logstash[30340]:         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:92) ~[?:?]

It seems like there is a problem with jruby and some missing file? Can anyone help?

I did try the config test and exit option and that seemed to work just fine.

2

It seems your .conf file is missing. Do you have good extension .conf?

3

I cannot help, just thought I would mention that it is trying to load a specific version of the concurrent gem, and it is not finding the version of that gem that it expects. I really don't know how to dig any deeper into that.

4

Which .conf file? I have several in /etc/logstash/conf.d/ that should be fine.

5

Is that something I can manually install or load? How can I see what gems are already installed?

6

logstash should come bundled with all the gems it depends on. On my system I see

drwxr-xr-x 4 root root  28 Mar  4  2020 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.5
drwxr-xr-x 4 root root  28 Sep  3  2020 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6
drwxr-xr-x 4 root root  28 Mar 10  2021 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.7
drwxr-xr-x 4 root root  28 Jun 14 20:17 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.8
drwxr-xr-x 4 root root 115 Oct 15 13:45 /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.9

Have you installed any additional plugins that could be adding dependencies?

7

[dwasss], have you try to test a single conf file from command line:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash/ -t
Then without "-t"
Have you enable debug mode?

  • config.debug: true
  • log.level: debug
8

I don't think I added any more plugins. I see that I do have the latest concurrent version there:

drwxr-xr-x   4 root root 4.0K Nov  2 19:19 concurrent-ruby-1.1.9
9

I tried enabling debug logging but logstash wouldn't even start so there wasn't really a point.

I did just try from the command line with the "-t" switch and got an error. I then tried running with "sudo" and it validated and exited OK. So I guess it's a permissions issue?

Using bundled JDK: /usr/share/logstash/jdk
/usr/share/logstash/logstash-core-plugin-api/lib/logstash-core-plugin-api/version.rb:22: warning: The Psych YAML extension failed to load.
Check your env for conflicting versions of SnakeYAML
See https://github.com/jruby/jruby/wiki/FAQs#why-does-the-psych-yaml-extension-fail-to-load-in-my-environment
[FATAL] 2021-11-03 17:40:50.633 [main] Logstash - Logstash stopped processing because of an error: (GemspecError) 
[!] There was an error while loading `logstash-core-plugin-api.gemspec`: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory. Bundler cannot continue.

 #  from /usr/share/logstash/logstash-core-plugin-api/logstash-core-plugin-api.gemspec:23
 #  -------------------------------------------
 #  
 >  require "logstash-core-plugin-api/version"
 #  
 #  -------------------------------------------

org.jruby.exceptions.StandardError: (GemspecError) 
[!] There was an error while loading `logstash-core-plugin-api.gemspec`: load error: psych -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory. Bundler cannot continue.
...
10

There were a lot of permissions issues I kept running into with various files, such as in the jruby plugins, java tmp folder, logstash logs folder, etc. that kept causing errors. Furthermore, the /usr/share/logstash/ directory and subfolders were all owned by root, thus there were more permissions issues. I just changed ownership to the logstash user and then things started working again.

I think I'm good now. Thanks @Rios and @Badger for your troubleshooting help. I'm not sure why upgrading from 7.12 to 7.15.1 jacked up the file and folder permissions :confounded:

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.