Hi,
I have a Logstash pipeline that parses events from multiple filebeat sources. Those filebeats are sending multiline XML files. After a month or so, it received an event that stops logstash due to xml parsing error (it seems like filebeat sent an incomplete xml, a separate issue i will troubleshoot later). I managed to find the page file that contains this particular xml event. The question is, how to resume Logstash or how to skip this event to make sure that Logstash will properly resume? Everytime i restart Logstash, it returns the same error from this exact xml. I enabled dead letter queue but seems like this event keeps blocking Logstash.
Thanks in advance.
Here's the error message:
> [2020-01-21T10:19:44,824][WARN ][logstash.filters.xml ][pipeline_name] Error parsing xml with XmlSimple {:source=>"grok_raw", :value=>"<ErrorInfo>\n<Sender>Gantry_1_L.PCBCamera</Sender>\n<File>f:\\b\\267\\src\\tpa\\station-sw\\rel\\711-rel\\src\\structcomps\\vision\\project\\activity.h</File>\n<Line>311</Line>\n<Function>GetDetailedError</Function>\n<UniqueId>852</UniqueId>\n<Severity>eERROR</Severity>\n<ErrorContext>eDEFAULT</ErrorContext>\n<Msg>\n<Context></Context>\n<Index>33208</Index>\n<Name>ActivityDetailedErrorInfo</Name>\n<Text>33208 For error tracking: activityID=%1, orderID=%2</Text>\n<IsDeveloperOnly>1</IsDeveloperOnly>\n<DisplaySender>1</DisplaySender>\n<Params>\n<Param>\n<Type>0</Type>\n<Value>403400001</Value>\n</Param>\n<Param>\n<Type>0</Type>\n<Value>0</Value>\n</Param>\n</Params>\n</Msg>\n<Time>\n<Year>2020</Year>\n<Month>1</Month>\n<Day>21</Day>", :exception=>#<REXML::ParseException: No close tag for /ErrorInfo/Time
> Line: 30
> Position: 699
> Last 80 unconsumed characters:
> >, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/treeparser.rb:28:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:288:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:45:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.0.7/lib/logstash/filters/xml.rb:185:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1800:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:250:in `block in start_workers'"]}
> [2020-01-21T10:19:45,555][ERROR][org.logstash.execution.WorkerLoop][pipeline_name] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
> java.lang.NullPointerException: null
> at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:651) ~[logstash-core.jar:?]
> at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:431) ~[logstash-core.jar:?]
> at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:422) ~[logstash-core.jar:?]
> at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$booleanCondition$4(EventCondition.java:140) ~[logstash-core.jar:?]
> at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:27) ~[logstash-core.jar:?]
> at org.logstash.generated.CompiledDataset42.compute(Unknown Source) ~[?:?]
> at org.logstash.generated.CompiledDataset43.compute(Unknown Source) ~[?:?]
> at org.logstash.generated.CompiledDataset47.compute(Unknown Source) ~[?:?]
> at org.logstash.generated.CompiledDataset48.compute(Unknown Source) ~[?:?]
> at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
> at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source) ~[?:?]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:425) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:292) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:28) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:90) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183) [jruby-complete-9.2.8.0.jar:?]
> at usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:250) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.runtime.Block.call(Block.java:129) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.8.0.jar:?]
> at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.8.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> [2020-01-21T10:19:45,882][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:250)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
> [2020-01-21T10:19:46,290][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
I have checked the contents of dead_letter_queue file
cat /usr/share/logstash/data/dead_lettter_queue/pipeline_name/1.log
but it's just this:
1
