HI guys,
I've two instances of ELK stack grepping the same log files containing XML payloads.
Flow is : input-plugin-file -> filter-plugin-grok -> filter-plugin-xml.
Log lines from files are parsed using grok and one of extracted field is parsed by xml plugin.
For old instance (ver. 6.3.1 of Elastisearch/Logstash/Kibana) - everything works fine.
New instance (ver. 7.1.0 of Elastisearch/Logstash/Kibana) - Logstash crashes occasionally with error like below:
Any ideas why ?
Thanks in advance !
[2019-08-09T10:37:57,360][ERROR][org.logstash.execution.WorkerLoop] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.
org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified.
at org.apache.xerces.dom.CoreDocumentImpl.checkQName(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.AttrNSImpl.setName(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.AttrNSImpl.rename(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.dom.CoreDocumentImpl.renameNode(Unknown Source) ~[xercesImpl.jar:?]
at nokogiri.internals.NokogiriHelpers.renameNode(NokogiriHelpers.java:792) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:374) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.removeNamespceRecursively(XmlDocument.java:381) ~[nokogiri.jar:?]
at nokogiri.XmlDocument.remove_namespaces(XmlDocument.java:356) ~[nokogiri.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_xml_minus_4_dot_0_dot_7.lib.logstash.filters.xml.RUBY$method$filter$0(/apps/elk/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.0.7/lib/logstash/filters/xml.rb:150) ~[?:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$method$do_filter$0(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$block$multi_filter$1(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:162) ~[?:?]
at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:146) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.BlockBody.yield(BlockBody.java:114) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.Block.yield(Block.java:165) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyArray.each(RubyArray.java:1792) ~[jruby-complete-9.2.7.0.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.filters.base.RUBY$method$multi_filter$0(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/filters/base.rb:159) ~[?:?]
at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:117) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:156) ~[jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:200) ~[jruby-complete-9.2.7.0.jar:?]
at org.logstash.config.ir.compiler.FilterDelegatorExt.doMultiFilter(FilterDelegatorExt.java:99) ~[logstash-core.jar:?]
at org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multiFilter(AbstractFilterDelegatorExt.java:115) ~[logstash-core.jar:?]
at org.logstash.generated.CompiledDataset12.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset13.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset14.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset15.compute(Unknown Source) ~[?:?]
at org.logstash.generated.CompiledDataset16.compute(Unknown Source) ~[?:?]
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64) [logstash-core.jar:?]
at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211]
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:440) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:304) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:36) [jruby-complete-9.2.7.0.jar:?]
at apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$2(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:235) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.runtime.Block.call(Block.java:124) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:295) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:274) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.RubyProc.call(RubyProc.java:270) [jruby-complete-9.2.7.0.jar:?]
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105) [jruby-complete-9.2.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
[2019-08-09T10:37:57,497][FATAL][logstash.runner ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.w3c.dom.DOMException: INVALID_CHARACTER_ERR: An invalid or illegal XML character is specified., :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "apps.elk.logstash_minus_7_dot_1_dot_0.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/apps/elk/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:235)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}