Logstash.runner: CompiledPipeline.java: java.lang.IllegalStateException: Error

usrjph · April 11, 2019, 1:40pm

I've updated our test box to 7.0, having deleted the old Elasticsearch database first and merged any config updates. I resolved a few config deprecations, including using _default in static mapping, but I'm now blocked by this error. Something to do with compiling a pipeline is crashing. I'd appreciate if someone could point me in the right direction, even if I just find out which configurations to tweak!

(Log truncated, error is huge!)

[2019-04-11T13:22:34,346] Converging pipelines state {:actions_count=>1}
[2019-04-11T13:22:34,359] Executing action {:action=>LogStash::PipelineAction::Create/pipeline_id:main}
[2019-04-11T13:22:50,935] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, :thread=>"#<Thread:0x371fc61c run>"}
[2019-04-11T13:22:51,264][FATAL][logstash.runner          ] h/config/ir/CompiledPipeline.java:332)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.lambda$compileDependencies$4(org/logstash/config/ir/CompiledPipeline.java:386)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.stream.ReferencePipeline$2$1.accept(java/util/stream/ReferencePipeline.java:175)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.Iterator.forEachRemaining(java/util/Iterator.java:116)", "java.util.Spliterators$IteratorSpliterator.forEachRemaining(java/util/Spliterators.java:1801)", "java.util.stream.AbstractPipeline.copyInto(java/util/stream/AbstractPipeline.java:481)", "java.util.stream.AbstractPipeline.wrapAndCopyInto(java/util/stream/AbstractPipeline.java:471)", "java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java/util/stream/ReduceOps.java:708)", "java.util.stream.AbstractPipeline.evaluate(java/util/stream/AbstractPipeline.java:234)", "java.util.stream.ReferencePipeline.collect(java/util/stream/ReferencePipeline.java:499)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.compileDependencies(org/logstash/config/ir/CompiledPipeline.java:400)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.flatten(org/logstash/config/ir/CompiledPipeline.java:361)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.split(org/logstash/config/ir/CompiledPipeline.java:332)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.lambda$compileDependencies$4(org/logstash/config/ir/CompiledPipeline.java:386)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.stream.ReferencePipeline$2$1.accept(java/util/stream/ReferencePipeline.java:175)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.Iterator.forEachRemaining(java/util/Iterator.java:116)", "java.util.Spliterators$IteratorSpliterator.forEachRemaining(java/util/Spliterators.java:1801)", "java.util.stream.AbstractPipeline.copyInto(java/util/stream/AbstractPipeline.java:481)", "java.util.stream.AbstractPipeline.wrapAndCopyInto(java/util/stream/AbstractPipeline.java:471)", "java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java/util/stream/ReduceOps.java:708)", "java.util.stream.AbstractPipeline.evaluate(java/util/stream/AbstractPipeline.java:234)", "java.util.stream.ReferencePipeline.collect(java/util/stream/ReferencePipeline.java:499)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.compileDependencies(org/logstash/config/ir/CompiledPipeline.java:400)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.flatten(org/logstash/config/ir/CompiledPipeline.java:361)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.filterDataset(org/logstash/config/ir/CompiledPipeline.java:287)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.lambda$compileDependencies$4(org/logstash/config/ir/CompiledPipeline.java:379)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.stream.ReferencePipeline$2$1.accept(java/util/stream/ReferencePipeline.java:175)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.Iterator.forEachRemaining(java/util/Iterator.java:116)", "java.util.Spliterators$IteratorSpliterator.forEachRemaining(java/util/Spliterators.java:1801)", "java.util.stream.AbstractPipeline.copyInto(java/util/stream/AbstractPipeline.java:481)", "java.util.stream.AbstractPipeline.wrapAndCopyInto(java/util/stream/AbstractPipeline.java:471)", "java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java/util/stream/ReduceOps.java:708)", "java.util.stream.AbstractPipeline.evaluate(java/util/stream/AbstractPipeline.java:234)", "java.util.stream.ReferencePipeline.collect(java/util/stream/ReferencePipeline.java:499)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.compileDependencies(org/logstash/config/ir/CompiledPipeline.java:400)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.flatten(org/logstash/config/ir/CompiledPipeline.java:361)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.filterDataset(org/logstash/config/ir/CompiledPipeline.java:287)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.lambda$compileDependencies$4(org/logstash/config/ir/CompiledPipeline.java:379)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.stream.ReferencePipeline$2$1.accept(java/util/stream/ReferencePipeline.java:175)", "java.util.stream.ReferencePipeline$3$1.accept(java/util/stream/ReferencePipeline.java:193)", "java.util.Iterator.forEachRemaining(java/util/Iterator.java:116)", "java.util.Spliterators$IteratorSpliterator.forEachRemaining(java/util/Spliterators.java:1801)", "java.util.stream.AbstractPipeline.copyInto(java/util/stream/AbstractPipeline.java:481)", "java.util.stream.AbstractPipeline.wrapAndCopyInto(java/util/stream/AbstractPipeline.java:471)", "java.util.stream.ReduceOps$ReduceOp.evaluateSequential(java/util/stream/ReduceOps.java:708)", "java.util.stream.AbstractPipeline.evaluate(java/util/stream/AbstractPipeline.java:234)", "java.util.stream.ReferencePipeline.collect(java/util/stream/ReferencePipeline.java:499)", "org.logstash.config.ir.CompiledPipeline$CompiledExecution.compileDependencies(org/logstash/config/ir/CompiledPipeline.java:400)",  ...
...
[2019-04-11T13:22:51,382] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

usrjph · April 12, 2019, 12:48pm

I found the stanzas at fault, I did so by re-introducing my config one section at a time until it crashed.

-    if [dest.ip] in ["localhost", "127.0.0.1", "::1"] { drop { } }
+    if ["localhost", "127.0.0.1", "::1"] in [dest.ip] { drop { } }

The check was backwards, checking for an array in a string. I believe in previous versions of Logstash this sort of thing produced a syntax error instead of Java vomit! The backwards line worked in 6.x.

Badger · April 12, 2019, 1:39pm

In 6.7 it does not get a syntax error. Not sure whether it does anything though . If you enable pipeline.java_execution (on by default in 7.0) then it crashes just like 7.0

system · May 10, 2019, 1:39pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.