Logstash 7.1.0 won't start

Elastic Stack Logstash
1

I had logstash 7.1.0 running on two servers last week, but today systemctl keeps restarting it. I su'd the logstash user to see what it's doing, and when starting it manually, I get this error:

[ERROR] 2019-09-30 15:25:26.498 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (RuntimeError)

    you might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command

load error: org/bouncycastle/bctls-jdk15on/1.61/bctls-jdk15on-1.61 -- java.lang.RuntimeException: BUG: we can not copy embedded jar to temp directory (LoadError)

I completely removed logstash and reinstalled, and keep getting the same error. I have validated that everything is owned by logstash.

Here are the logs I have:

[2019-09-30T16:49:17,730][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-09-30T16:49:24,954][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[2019-09-30T16:49:24,998][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x7afd4d5b run>"}
[2019-09-30T16:49:25,298][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-09-30T16:49:25,319][INFO ][logstash.inputs.http ] Starting http input listener {:address=>"0.0.0.0:8080", :ssl=>"false"}
[2019-09-30T16:49:25,482][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:4501"}
[2019-09-30T16:49:25,494][ERROR][org.logstash.Logstash ] java.lang.NoClassDefFoundError: Could not initialize class org.logstash.config.ir.compiler.ComputeStepSyntaxElement
[2019-09-30T16:49:43,589][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.0"}
[2019-09-30T16:49:50,613][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2019-09-30T16:49:50,824][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-geoip-6.0.1-java/vendor/GeoLite2-City.mmdb"}
[2019-09-30T16:49:50,879][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x32d7a129 run>"}
[2019-09-30T16:49:51,221][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-09-30T16:49:51,263][ERROR][org.logstash.Logstash ] java.lang.ExceptionInInitializerError

2

I had the same error message today after Logstash running properly for a long time, but I could solve it. Maybe my findings are a help for you too.

First, after search for answers, I found this issue on github, which probably gave me the right advice.

I've then checked my /tmp directory and found a lot of logstash related folders in there, so it seems to me, that I also had the problem, that these folders never got deleted.

I followed the solution in the issue, but that was no help in the beginning, I had to remove all the logstash folders in /tmp. Then Logstash started again.

I don't know if the number of files in the /tmp directory is related to the LS_OPEN_FILES value in the /etc/default/logstash file and that caused the error, maybe somebody here knows.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.