Hi there, I ran a logstash instance from April and it was working. For some reason now it has stopped. Restarting all ELK services doesn't solve the problem. Is there a way to troubleshoot this? What are the steps? To make sure log data is coming in tcpdump shows UDP 5514 syslog. When I look at the indices it also stopped as from 11th of April.
Have you looked in the Logstash log file, possibly while raising the log level to get additional details?
I think this is it.. :
An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable
however permissions look good on that dir..
still..:
An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in block in validate_all'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in'"]}
[2018-06-13T15:47:29,142][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
What's the output of ls -l /usr/share/logstash/data/queue? Which user does Logstash run as?
total 0
drwxrwxr-x. 2 root root 6 Apr 11 11:06 .
drwxrwxr-x. 4 logstash logstash 69 Apr 11 11:08 ..
logstash
Logstash runs like this:
[Service]
Type=simple
User=logstash
Group=logstash
Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
Prefixing the path with '-' makes it try to load, but if the file doesn't
exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "--debug"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384
Some improvement..:
] --------------- Logstash Settings -------------------
[2018-06-13T16:48:59,222][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-06-13T16:48:59,241][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException: /usr/share/logstash/data/.lock, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:84)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:102)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:107)", "sun.nio.fs.UnixFileSystemProvider.newFileChannel(sun/nio/fs/UnixFileSystemProvider.java:177)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:287)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:335)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:75)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:468)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:370)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:335)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219)", "RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132)", "usr.share.logstash.lib.bootstrap.environment.invokeOther55:run(usr/share/logstash/lib/bootstrap//usr/share/logstash/lib/bootstrap/environment.rb:67)", "usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:67)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:627)", "org.jruby.Ruby.runScript(org/jruby/Ruby.java:828)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:747)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:765)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:578)", "org.logstash.Logstash.run(org/logstash/Logstash.java:81)", "org.logstash.Logstash.main(org/logstash/Logstash.java:45)"]}
[2018-06-13T16:48:59,247][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit
Indices status:
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .monitoring-es-6-2018.04.11 GWH5uCuOQjGjdSkf22ZoUw 1 0 1839 12 1mb 1mb
yellow open syslog-2018.12.22 ULJz3m2jSxS7kFrcJPsQXQ 5 1 1 0 13.8kb 13.8kb
yellow open logstash-2018.12.22 glSsUzStTrOcL3u8U0TxtA 5 1 1 0 12.1kb 12.1kb
green open .watches n0HQZ2pKT4GTvfMHrOeo2g 1 0 6 0 32.9kb 32.9kb
green open .monitoring-alerts-6 whBL9bysR7au2_1bBSMtPQ 1 0 1 0 6.1kb 6.1kb
yellow open logstash-2018.12.23 gJgTS6i5SSeYa8mlK7eO7g 5 1 3 0 36.2kb 36.2kb
green open .security-6 ZXbp_DODSouFZamobe3Wdg 1 0 3 0 9.8kb 9.8kb
green open .kibana FtIuYpWUSV-pZ78VskTTnw 1 0 2 0 11.2kb 11.2kb
yellow open syslog-2018.04.11 OyXhG8wjT9i3kPjGCU24Lw 5 1 5 0 25.4kb 25.4kb
close .watcher-history-7-2018.04.11 01Z34Tk8SoCWrrQj_oFlYA
green open .triggered_watches dm4mpxy_Q4GTwLJPVjQ7ng 1 0 0 0 15.5kb 15.5kb
yellow open syslog-2018.12.23 PtgRZgsQTC-z3BIBYPTPqg 5 1 3 0 28.7kb 28.7kb
yellow open logstash-2018.04.11 yXMKHnulSHOimPlNwsgUXw 5 1 5 0 22.7kb 22.7kb
Looks like it works 
Now find a way to parse the data and view it in Kibana 
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.