Logstash crash with FATAL error

Elastic Stack Logstash
1

something in either my data or my logstash is killing logstash with following error.

I have try this multiple time but I can't figure out what record does it gets kill on.

how do I do the debug on this one?
I am running this from command like to test it.

 [[main]>worker0] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.lang.NullPointerException: null
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset26.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset27.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset28.compute(Unknown Source) ~[?:?]

[FATAL] 2020-12-16 15:20:52.287 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:832)"]}

2

Is there more to the log?

3

That looks a lot like this -- doing a numeric conditional on a field that does exist.

1 Like
4

it is huge log on stdout. here it is

[ERROR] 2020-12-16 15:20:52.083 [[main]>worker0] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.lang.NullPointerException: null
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset39.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset46.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset47.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset48.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset49.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset50.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset52.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset53.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset54.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset55.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset56.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset57.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset60.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset61.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset62.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset63.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset64.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset67.compute(Unknown Source) ~[?:?]
        at org.logstash.config.ir.compiler.DatasetCompiler$Complement.compute(DatasetCompiler.java:495) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset68.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset69.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset137.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset138.compute(Unknown Source) ~[?:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:321) ~[logstash-core.jar:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:311) ~[logstash-core.jar:?]
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83) [logstash-core.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.11.1.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.Block.call(Block.java:139) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:318) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.11.1.jar:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
warning: thread "[main]>worker0" terminated with exception (report_on_exception is true):
java.lang.IllegalStateException: java.lang.NullPointerException
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
        at java.lang.Thread.run(java/lang/Thread.java:832)
Caused by: java.lang.NullPointerException
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162)
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47)
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset57.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset60.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset61.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset62.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset63.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset64.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset67.compute(Unknown Source)
        at org.logstash.config.ir.compiler.DatasetCompiler$Complement.compute(DatasetCompiler.java:495)
        at org.logstash.generated.CompiledDataset68.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset69.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset70.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset71.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset72.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset133.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset134.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset135.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset136.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset137.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset138.compute(Unknown Source)
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:321)
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:311)
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86)
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        at org.jruby.runtime.Block.call(Block.java:139)
        at org.jruby.RubyProc.call(RubyProc.java:318)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.base/java.lang.Thread.run(Thread.java:832)
[FATAL] 2020-12-16 15:20:52.287 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:832)"]}
[ERROR] 2020-12-16 15:20:52.309 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
5

Badger,
How do I find out which one is doing this?

6

I do not think you can tell. If there is any chance that the field will not exist you need to test whether it exists before doing a numeric comparison to it.

7

ok. I am doing this for almost all field which I do comparison.

this means I have to test out one by one and find out which one is doing this.

8

Found it. it took me many hour. just break down long code to small chunk.

but most time it took to find out which record it is failing and then nail down what it was.

one of the field I was creating using ruby. but it was not getting created in one case. and then on bottom of the code I was testing that value. Thank you both of you

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.