# Logstash crash with FATAL error

**URL:** https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919
**Category:** Logstash
**Created:** [December 16, 2020, 10:21pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919 "2020-12-16T22:21:40Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 16, 2020, 10:21pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/1 "2020-12-16T22:21:40Z")

</div>

something in either my data or my logstash is killing logstash with following error.

I have try this multiple time but I can't figure out what record does it gets kill on.

how do I do the debug on this one?  
I am running this from command like to test it.

```
 [[main]>worker0] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.lang.NullPointerException: null
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset26.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset27.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset28.compute(Unknown Source) ~[?:?]

```

`[FATAL] 2020-12-16 15:20:52.287 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:832)"]}`

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 16, 2020, 10:25pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/2 "2020-12-16T22:25:02Z")

</div>

Is there more to the log?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 16, 2020, 10:26pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/3 "2020-12-16T22:26:56Z")

</div>

That looks a lot like [this](https://discuss.elastic.co/t/comparing-number-in-conditionnal-event-treatement/217043/5) -- doing a numeric conditional on a field that does exist.

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 16, 2020, 10:28pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/4 "2020-12-16T22:28:09Z")

</div>

it is huge log on stdout. here it is

```
[ERROR] 2020-12-16 15:20:52.083 [[main]>worker0] WorkerLoop - Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
java.lang.NullPointerException: null
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162) ~[logstash-core.jar:?]
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset39.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset46.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset47.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset48.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset49.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset50.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset52.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset53.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset54.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset55.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset56.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset57.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset60.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset61.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset62.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset63.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset64.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset67.compute(Unknown Source) ~[?:?]
        at org.logstash.config.ir.compiler.DatasetCompiler$Complement.compute(DatasetCompiler.java:495) ~[logstash-core.jar:?]
        at org.logstash.generated.CompiledDataset68.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset69.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset137.compute(Unknown Source) ~[?:?]
        at org.logstash.generated.CompiledDataset138.compute(Unknown Source) ~[?:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:321) ~[logstash-core.jar:?]
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:311) ~[logstash-core.jar:?]
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83) [logstash-core.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:564) ~[?:?]
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207) [jruby-complete-9.2.11.1.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.runtime.Block.call(Block.java:139) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.RubyProc.call(RubyProc.java:318) [jruby-complete-9.2.11.1.jar:?]
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.11.1.jar:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
warning: thread "[main]>worker0" terminated with exception (report_on_exception is true):
java.lang.IllegalStateException: java.lang.NullPointerException
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)
        at java.lang.Thread.run(java/lang/Thread.java:832)
Caused by: java.lang.NullPointerException
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:679)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:453)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:444)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:162)
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:47)
        at org.logstash.generated.CompiledDataset22.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset24.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset57.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset60.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset61.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset62.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset63.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset64.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset67.compute(Unknown Source)
        at org.logstash.config.ir.compiler.DatasetCompiler$Complement.compute(DatasetCompiler.java:495)
        at org.logstash.generated.CompiledDataset68.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset69.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset70.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset71.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset72.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset133.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset134.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset135.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset136.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset137.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset138.compute(Unknown Source)
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:321)
        at org.logstash.config.ir.CompiledPipeline$CompiledOrderedExecution.compute(CompiledPipeline.java:311)
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:83)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:426)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:293)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:24)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:86)
        at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)
        at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)
        at org.jruby.runtime.Block.call(Block.java:139)
        at org.jruby.RubyProc.call(RubyProc.java:318)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.base/java.lang.Thread.run(Thread.java:832)
[FATAL] 2020-12-16 15:20:52.287 [LogStash::Runner] runner - An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:105)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:564)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:426)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:293)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:278)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:318)", "java.lang.Thread.run(java/lang/Thread.java:832)"]}
[ERROR] 2020-12-16 15:20:52.309 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
```

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 16, 2020, 10:31pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/5 "2020-12-16T22:31:16Z")

</div>

Badger,  
How do I find out which one is doing this?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [December 16, 2020, 10:41pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/6 "2020-12-16T22:41:17Z")

</div>

I do not think you can tell. If there is any chance that the field will not exist you need to test whether it exists before doing a numeric comparison to it.

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 16, 2020, 10:46pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/7 "2020-12-16T22:46:41Z")

</div>

ok. I am doing this for almost all field which I do comparison.

this means I have to test out one by one and find out which one is doing this.

---

<div class="post-metadata">

### Author: ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)
#### Post date: [December 17, 2020, 4:50pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/8 "2020-12-17T16:50:08Z")

</div>

Found it. it took me many hour. just break down long code to small chunk.

but most time it took to find out which record it is failing and then nail down what it was.

one of the field I was creating using ruby. but it was not getting created in one case. and then on bottom of the code I was testing that value. Thank you both of you

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 14, 2021, 4:50pm UTC](https://discuss.elastic.co/t/logstash-crash-with-fatal-error/258919/9 "2021-01-14T16:50:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
