HI Team,
Could you please consider adding setting "tag_on_failure" for xml filter plugin as it is done for grok filter ?
It could add a tag (for example "_xmlfailure") in case of XML parsing error or when plugin crashes for any other reason.
Would be very helpful.
Thanks !
Ged
The xml filter will add "_xmlparsefailure" if a parse failure occurs. The value is not configurable.
Thanks Badger !
Will it also work in case of error shown below ? Would be great to catch such an exception instead of crashing Logstash. Looks like this one occurs on removing namespaces.
[2019-08-09T10:37:57,360][ERROR][org.logstash.execution.WorkerLoop] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified.
at org.apache.xerces.dom.CoreDocumentImpl.checkQName(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.AttrNSImpl.setName(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.AttrNSImpl.rename(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.CoreDocumentImpl.renameNode(Unknown Source) ~[xercesImpl.jar:?]
at nokogiri.internals.NokogiriHelpers.renameNode(NokogiriHelpers.java:792) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:374) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.remove_namespaces(XmlDocument.java:356) ~[nokogiri.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_xml_minus_4_dot_0_dot_7.lib.logstash.filters.xml.RUBY$method$filter$0(/apps/elk/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.0.7/lib/logstash/filters/xml.rb:150) ~[?:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?]
at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.BlockBody.yield(BlockBody.java:114) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.Block.yield(Block.java:165) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyArray.each(RubyArray.java:1792) ~[jruby-complete-9.2.7.0.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) ~[jruby-complete-9.2.7.0.jar:?]
at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:99) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?]
at org.logstash.generated.CompiledDataset12.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset13.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset14.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset15.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset16.compute(Unknown Source) ~[?:?]
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.7.0.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:235) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
[2019-08-09T10:37:57,497][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified., :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:235)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}I am really not sure. You would have to try it.
Looks like not, so would be nice to catch such exceptions and add also add tag in such cases to avoid Logstash crashing.
Ged
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.