Logstash fails to run if data.path is a symlink

I host /var/lib/logstash on a separate partition, this directory is a symlink elsewhere.

Logstash fails to start with the following error message in logs:

[2018-06-05T13:12:24,295][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-06-05T13:12:24,300][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-06-05T13:12:24,456][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/modules/arcsight/configuration"}
[2018-06-05T13:12:24,674][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.nio.file.FileAlreadyExistsException: /var/lib/logstash, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:88)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:102)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:107)", "sun.nio.fs.UnixFileSystemProvider.createDirectory(sun/nio/fs/UnixFileSystemProvider.java:384)", "java.nio.file.Files.createDirectory(java/nio/file/Files.java:674)", "java.nio.file.Files.createAndCheckIsDirectory(java/nio/file/Files.java:781)", "java.nio.file.Files.createDirectories(java/nio/file/Files.java:727)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:57)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:468)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:370)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:335)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)", "usr.share.logstash.lib.bootstrap.environment.invokeOther55:run(usr/share/logstash/lib/bootstrap//usr/share/logstash/lib/bootstrap/environment.rb:67)", "usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:67)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:828)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:747)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:765)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:578)", "org.logstash.Logstash.run(org/logstash/Logstash.java:81)", "org.logstash.Logstash.main(org/logstash/Logstash.java:45)"]}
[2018-06-05T13:12:24,677][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

Feels like we fail when trying to create a lock file, because logstash uses Files.createDirectories(and it does not follow symlinks):

This issue was already highlighted there without any response: Startup fails if path.data is a symlink

Can we please add additional check if data.dir already exist before
calling Files.createDirectories?

I've merged your fix below for this. Thanks for your contribution!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.