I host /var/lib/logstash on a separate partition, this directory is a symlink elsewhere.
Logstash fails to start with the following error message in logs:
[2018-06-05T13:12:24,295][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2018-06-05T13:12:24,300][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2018-06-05T13:12:24,456][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"arcsight", :directory=>"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/x-pack-6.2.4-java/modules/arcsight/configuration"}
[2018-06-05T13:12:24,674][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.nio.file.FileAlreadyExistsException: /var/lib/logstash, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:88)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:102)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:107)", "sun.nio.fs.UnixFileSystemProvider.createDirectory(sun/nio/fs/UnixFileSystemProvider.java:384)", "java.nio.file.Files.createDirectory(java/nio/file/Files.java:674)", "java.nio.file.Files.createAndCheckIsDirectory(java/nio/file/Files.java:781)", "java.nio.file.Files.createDirectories(java/nio/file/Files.java:727)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:57)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:468)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:370)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:335)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)", "usr.share.logstash.lib.bootstrap.environment.invokeOther55:run(usr/share/logstash/lib/bootstrap//usr/share/logstash/lib/bootstrap/environment.rb:67)", "usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:67)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:828)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:747)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:765)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:578)", "org.logstash.Logstash.run(org/logstash/Logstash.java:81)", "org.logstash.Logstash.main(org/logstash/Logstash.java:45)"]}
[2018-06-05T13:12:24,677][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
Feels like we fail when trying to create a lock file, because logstash uses Files.createDirectories(and it does not follow symlinks):
This issue was already highlighted there without any response: Startup fails if path.data is a symlink
Can we please add additional check if data.dir already exist before
calling Files.createDirectories?