I had logstash 7.1.0 running on two servers last week, but today systemctl keeps restarting it. I su'd the logstash user to see what it's doing, and when starting it manually, I get this error:
[ERROR] 2019-09-30 15:25:26.498 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (RuntimeError)
you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command
load error: org/bouncycastle/bctls-jdk15on/1.61/bctls-jdk15on-1.61 -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory (LoadError)
I completely removed logstash and reinstalled, and keep getting the same error. I have validated that everything is owned by logstash.
Here are the logs I have:
[2019-09-30T16:49:17,730][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-09-30T16:49:24,954][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[2019-09-30T16:49:24,998][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x7afd4d5b run>"}
[2019-09-30T16:49:25,298][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-09-30T16:49:25,319][INFO ][logstash.inputs.http ] Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
[2019-09-30T16:49:25,482][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:4501"}
[2019-09-30T16:49:25,494][ERROR][org.logstash.Logstash ] java.lang.NoClassDefFoundError: Could not initialize class org.logstash.config.ir.compiler.ComputeStepSyntaxElement
[2019-09-30T16:49:43,589][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-09-30T16:49:50,613][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2019-09-30T16:49:50,824][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[2019-09-30T16:49:50,879][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x32d7a129 run>"}
[2019-09-30T16:49:51,221][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-09-30T16:49:51,263][ERROR][org.logstash.Logstash ] java.lang.ExceptionInInitializerError