Sep 27 17:22:55 ldnlinsocelk01 systemd[1]: Started logstash.
Sep 27 17:22:55 ldnlinsocelk01 logstash[15386]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sep 27 17:22:57 ldnlinsocelk01 logstash[15386]: WARNING: An illegal reflective access operation has occurred
Sep 27 17:22:57 ldnlinsocelk01 logstash[15386]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
Sep 27 17:22:57 ldnlinsocelk01 logstash[15386]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
Sep 27 17:22:57 ldnlinsocelk01 logstash[15386]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Sep 27 17:22:57 ldnlinsocelk01 logstash[15386]: WARNING: All illegal access operations will be denied in a future release
Sep 27 17:23:11 ldnlinsocelk01 logstash[15386]: Thread.exclusive is deprecated, use Thread::Mutex
Sep 27 17:23:15 ldnlinsocelk01 logstash[15386]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Sep 27 17:23:15 ldnlinsocelk01 logstash[15386]: [2019-09-27T17:23:15,628][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/var/lib/logstash" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:489:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:271:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:182:in block in validate_all'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:181:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in'"]}
Sep 27 17:23:15 ldnlinsocelk01 logstash[15386]: [2019-09-27T17:23:15,664][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Sep 27 17:23:15 ldnlinsocelk01 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Sep 27 17:23:15 ldnlinsocelk01 systemd[1]: logstash.service: Failed with result 'exit-code'.
Sep 27 17:23:16 ldnlinsocelk01 systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Sep 27 17:23:16 ldnlinsocelk01 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 29.
Sep 27 17:23:16 ldnlinsocelk01 systemd[1]: Stopped logstash.
this is after removing and reinstalling logstash several times today.
ran the install from a fresh downloaded .deb on ubuntu 18.04 which for the past three weeks or so has been quite happily running elk stack in a test environment.
today after finally sorting out a filter problem, i was running through the system checking that everything was running ok before changing data paths to ingest new logs when i noticed the logstash service wasn't processing any filebeats inputs.
i restarted logstash and then the problems started.
i've completely cleaned the system of all references to logstash after the third install, cleared the synaptics cach of the downloaded install, redownloaded a fresh logstash from the elastic.co site, ran the installs, installed and checked permissions on every logstash directory i can find
/etc/logstash
/usr/share/logstash
/lib/share/logstash
and all the other little places it drops the files......
ran with the default permissions group for logstash, changed it to root, added logstash to root, added root to logstash, followed the errors on the access denies, to conclusion and i still can't get this to work.
can someone cast their eye over this and tell me if it seems familiar? has anyone had this sort of issue on a fresh install, repeated install or otherwise?
