Logstash installation problems Ubuntu 20.04 VM

Could someone please walk me through the correct Logstash installation procedure for Ubuntu 20.04? Starting with correct JDK installation as required by Logstash installation.

Elasticsearch and Kibana install just fine for me.

But I can't get a successful Logstash installation. I've tried several times. I must be doing something wrong, it can't be this hard.

From https://www.elastic.co/guide/en/logstash/current/installing-logstash.html, I'm taking these instructions literally:

  1. Logstash installation must use JDK version 8 or 11, nothing subsequent to 11?
  2. Logstash installation cannot use the JDK bundled with Elasticsearch?

So I dutifully install JDK 11 (see below)
When I then install Logstash I get all kinds of "reflective access" warnings. Then fatal error during execution (different ones, I've been through this multiple times).

I've tried it with several different Java 11 JDKs. Oracle commercial (jdk-11.0.7_linux-x64_bin) and Oracle open (openjdk-11+28_linux-x64_bin). Manual JAVA_HOME, and PATH to binary. Also tried Ubuntu's openjdk-11-jdk (auto JAVA_HOME and PATH to binary). Even tried a special Ubuntu Oracle installer (auto JAVA_HOME and PATH to binary).

Thank you

Posting those would be helpful :slight_smile:

Thanks Mark. Here is one example from install, one from execution.

Unpacking logstash (1:7.7.0-1) ...
Setting up logstash (1:7.7.0-1) ...
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
bin/logstash -e
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[FATAL] 2020-06-01 00:28:04.975 [main] runner - An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:528:in `validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:288:in `validate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:199:in `block in validate_all'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:198:in `validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:305:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:263:in `run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:88:in `<main>'"]}

Can you look into that further?

drwxrwxr-x 2 logstash 4096 Jun 1 13:20 data/
Weird, since I hadn't mucked with permissions at all. It was a brand new VM.

So I fixed that. Had to re-install Logstash (I had since uninstalled it). This time the installation only complained about one thing: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated

But different execution error

./logstash -e
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2020-06-01 20:46:11.823 [main] writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2020-06-01 20:46:11.842 [main] writabledirectory - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
  bin/logstash -f CONFIG_PATH [-t] [-r] [] [-w COUNT] [-l LOG]
  bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
  bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
  bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
  bin/logstash --help
[ERROR] 2020-06-01 20:46:12.148 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

I checked and there is no /usr/share/logstash/config directory at all. I'm guessing there should be something there by default?

There should be, yes. Did the install process work?

Yes, the install process seemed to work:

sudo apt-get install logstash
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  gsfonts gsfonts-x11
Use 'sudo apt autoremove' to remove them.
The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 29 not upgraded.
Need to get 0 B/167 MB of archives.
After this operation, 295 MB of additional disk space will be used.
Selecting previously unselected package logstash.
(Reading database ... 350275 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a7.7.0-1_all.deb ...
Unpacking logstash (1:7.7.0-1) ...
Setting up logstash (1:7.7.0-1) ...
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash

Can you try an apt-get purge logstash and then a reinstall?

Yes, the results pasted above were after a purge, I had confirmed the non-existence of /usr/share/logstash, then reinstall. I just performed the cycle again, same results. Only the one Fixnum warning during installation. Afterwards (per your diagnosis) no /usr/share/logstash/config. No o=w permission for /usr/share/logstash/data.

I wonder if anyone could please try to install Logstash in Ubuntu 20.04 LTS, and let me know if it works correctly, or fails, similar to my attempts?

I don't think it's jdk related, I've tried several. I don't know if there are more installation failures in addition to the first two we've noticed: no config directory and no "other" write permissions on anything in /usr/share/logstash/. Here's what that directory looks like for me post-install.

ls -fla /usr/share/logstash
total 688
drwxrwxr-x   9 logstash logstash   4096 Jun  3 13:45 x-pack
drwxrwxr-x   3 logstash logstash   4096 Jun  3 13:45 tools
drwxrwxr-x   4 logstash logstash   4096 Jun  3 13:45 logstash-core
drwxrwxr-x   4 logstash logstash   4096 Jun  3 13:45 vendor
drwxrwxr-x  11 logstash logstash   4096 Jun  3 13:45 .
-rw-r--r--   1 logstash logstash   2276 May 28 11:38 CONTRIBUTORS
drwxrwxr-x   2 logstash logstash   4096 Jun  3 13:45 bin
drwxrwxr-x   3 logstash logstash   4096 Jun  3 13:45 logstash-core-plugin-api
drwxrwxr-x   2 logstash logstash   4096 May 28 11:41 data
-rw-r--r--   1 logstash logstash 601073 May 28 11:38 NOTICE.TXT
drwxr-xr-x 255 root     root      12288 Jun  3 13:45 ..
drwxrwxr-x   6 logstash logstash   4096 Jun  3 13:45 lib
-rw-r--r--   1 logstash logstash   4041 May 28 11:39 Gemfile
drwxrwxr-x   4 logstash logstash   4096 Jun  3 13:45 modules
-rw-r--r--   1 logstash logstash  22684 May 28 11:40 Gemfile.lock
-rw-r--r--   1 logstash logstash  13675 May 28 11:38 LICENSE.txt

And here's the installer output

sudo apt-get install logstash
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 42 not upgraded.
Need to get 167 MB of archives.
After this operation, 295 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 logstash all 1:7.7.1-1 [167 MB]
Fetched 167 MB in 6s (27.5 MB/s)                                               
Selecting previously unselected package logstash.
(Reading database ... 351356 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a7.7.1-1_all.deb ...
Unpacking logstash (1:7.7.1-1) ...
Setting up logstash (1:7.7.1-1) ...
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash

Thanks for the help

Hello,

I believe I managed to reproduce the cause of your issue - the warnings initially seem to be a bit of a red herring (they're warnings, not errors, right?) which is how I found your post :slight_smile:

I'm unable to install logstash on a brand new Ubuntu 18.04 VM myself, using AdoptOpenJDK 11.0.7+10. The initial apt error is a bit cryptic:

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 52 not upgraded.
Need to get 167 MB of archives.
After this operation, 295 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 logstash all 1:7.7.1-1 [167 MB]
Fetched 167 MB in 4s (44.4 MB/s)
Selecting previously unselected package logstash.
(Reading database ... 82148 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a7.7.1-1_all.deb ...
Unpacking logstash (1:7.7.1-1) ...
Setting up logstash (1:7.7.1-1) ...
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2020-06-11T15:41:06.296Z [main] WARN FilenoUtil : Native subprocess control requires open access to sun.nio.ch
Pass '--add-opens java.base/sun.nio.ch=org.jruby.dist' or '=org.jruby.core' to enable.
Errno::EBADF: Bad file descriptor - systemctl
            spawn at org/jruby/RubyProcess.java:1635
            spawn at org/jruby/RubyKernel.java:1658
        popen_run at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:202
           popen3 at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:98
          execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/detector.rb:74
   detect_systemd at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/detector.rb:29
  detect_platform at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/detector.rb:24
           detect at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/detector.rb:18
   setup_defaults at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/cli.rb:153
          execute at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/cli.rb:119
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/cli.rb:114
              run at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132
           <main> at /usr/share/logstash/lib/systeminstall/pleasewrap.rb:28
Unable to install system startup script for Logstash.
chmod: cannot access '/etc/default/logstash': No such file or directory
dpkg: error processing package logstash (--configure):
 installed logstash package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 logstash
W: --force-yes is deprecated, use one of the options starting with --allow instead.
E: Sub-process /usr/bin/dpkg returned an error code (1)

The problem seems to actually be https://github.com/jruby/jruby/issues/5766 because using the test provided in that issue I got the following:

ubuntu@ip-10-62-220-120:~$ ruby -r open3 -e 'Open3.popen3("ls")'
ubuntu@ip-10-62-220-120:~$ /usr/share/logstash/vendor/jruby/bin/jruby -r open3 -e 'Open3.popen3("ls")'
2020-06-11T16:06:36.824Z [main] WARN FilenoUtil : Native subprocess control requires open access to sun.nio.ch
Pass '--add-opens java.base/sun.nio.ch=org.jruby.dist' or '=org.jruby.core' to enable.
Errno::EBADF: Bad file descriptor - ls
      spawn at org/jruby/RubyProcess.java:1635
      spawn at org/jruby/RubyKernel.java:1658
  popen_run at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:202
     popen3 at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:98
     <main> at -e:1

So the issue failing the install is actually a jruby issue preventing open3.popen3 from working :confused:
Rather confusingly, following the warning (not error!) about reflections doesn't help all that much:

ubuntu@ip-10-62-220-120:~$ JAVA_OPTS='--add-opens java.base/sun.nio.ch=org.jruby.dist' /usr/share/logstash/vendor/jruby/bin/jruby -r open3 -e 'Open3.popen3("ls")'
Errno::EBADF: Bad file descriptor - ls
      spawn at org/jruby/RubyProcess.java:1635
      spawn at org/jruby/RubyKernel.java:1658
  popen_run at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:202
     popen3 at /usr/share/logstash/vendor/jruby/lib/ruby/stdlib/open3.rb:98
     <main> at -e:1

but digging into the endless jruby issues on GitHub provides some magic that does fix this:

ubuntu@ip-10-62-220-120:~$ JAVA_OPTS='--add-opens java.base/sun.nio.ch=org.jruby.dist  --add-opens java.base/java.io=org.jruby.dist' /usr/share/logstash/vendor/jruby/bin/jruby -r open3 -e 'Open3.popen3("ls")'

Whew! So far, so good, right? The only way I've managed to get this install to work is by taking the startup.options file from the logstash package, adding a new line with JAVA_OPTS='--add-opens java.base/sun.nio.ch=org.jruby.dist --add-opens java.base/java.io=org.jruby.dist' and then installing logstash ensuring that apt does not overwrite the startup.options file :roll_eyes: Logstash does not need this anymore but the pleaserun installer seems to.

Even more confusingly, when you add the --add-opens environment variable to that file, the shell script attempts to evaluate the arguments as variables (because they are in the form of foo=bar) and throws a shell substitution error which send you down another rabbit hole entirely if you run apt-get manually in hopes of figuring this out :sweat_smile: I've been struggling with this for a good few hours now and still haven't managed to successfully install using ansible but when I drop into a shell everything seems to be a-ok:

ubuntu@ip-10-62-220-114:~$ sudo apt-get -y -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold install logstash=1:7.7.1-1
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  logstash
0 upgraded, 1 newly installed, 0 to remove and 54 not upgraded.
Need to get 167 MB of archives.
After this operation, 295 MB of additional disk space will be used.
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 logstash all 1:7.7.1-1 [167 MB]
Fetched 167 MB in 3s (53.2 MB/s)
Selecting previously unselected package logstash.
(Reading database ... 82148 files and directories currently installed.)
Preparing to unpack .../logstash_1%3a7.7.1-1_all.deb ...
Unpacking logstash (1:7.7.1-1) ...
Setting up logstash (1:7.7.1-1) ...

Configuration file '/etc/logstash/startup.options'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
 ==> Keeping old config file as default.
Using provided startup.options file: /etc/logstash/startup.options
/usr/share/logstash/bin/system-install: line 56: java.base/sun.nio.ch: bad substitution
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.31/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
ubuntu@ip-10-62-220-114:~$ echo $?
0

Hope this helps...

@warkolm is this something that can be fixed in the debian package itself? Seems to me like the --add-opens options are required for pleaserun with the version of jruby provided with logstash...

Thanks,
Simon

edit: I've still not managed to figure out why the installation fails when orchestrated by ansible but I had to resort to pre-populating /etc/logstash/defaults before installing the package and calling /usr/share/logstash/bin/system-install /etc/logstash/startup.options after the package is installed - it's rather odd (and I don't have the time to dig into this further at the moment) but it does work.

edit 2: after a lot of further debugging I think I've got this figured out now. The /usr/share/logstash/bin/system-install: line 56: java.base/sun.nio.ch: bad substitution error is actually fatal when installing from Ansible - not sure why that is the case but this is what was causing my Ansible woes. My final working solution is to add the following to /etc/logstash/startup.options before installing logstash:

JAVA_OPTS="@/etc/logstash/java_opts"

and create a /etc/logstash/java_opts file (also prior to the installation) with the following content:

--add-opens java.base/sun.nio.ch=org.jruby.dist
--add-opens java.base/java.io=org.jruby.dist

:tada:

I've really lost track of all the settings mentioned here. I just wanted to say that I've installed Logstash on Ubuntu 20.04 lately and while there are some warning, it seems to be running without problems. All I did was

sudo apt-get install default-jre
sudo apt-get update && sudo apt-get install logstash

And then I set the variables in /etc/environment to

LS_HOME=/usr/share/logstash
LS_SETTINGS_DIR=/etc/logstash

I don't know if this is helpful to you, but this way my installation could find its configuration.

Wow @simons, fantastic work! Lacking your expertise, I naively figured I must have been doing it wrong. I had tried several clean distros, five jdks, and both the deb and zip installation procedures.

I wonder if the Logstash installer couldn't be made to be entirely self-sufficient. (It can always look to Elasticsearch and Kibana for inspiration :slight_smile:)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.