# Console XML works, but not file XML **URL:** \ **Category:** Logstash\ **Created:** [November 5, 2019, 4:36pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645 "2019-11-05T16:36:17Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![kcb](https://avatars.discourse-cdn.com/v4/letter/k/da6949/32.png) [@kcb](https://discuss.elastic.co/u/kcb)\ **Post date:** [November 5, 2019, 4:36pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/1 "2019-11-05T16:36:17Z")
We have written a logstash script to process an XMLfile and export contents to stdout{}. We have verified the script works when the input is specified as stdin{} However, when we specify a file as the input, the XML does not process as expected and we get errors like: Error parsing xml with XmlSimple Config file contents are: input { file { path =\> "D:/logstash-7.4.1/conf/geoserver\_audit\_20191030\_2.log" sincedb\_path =\> "nul" start\_position =\> "beginning" type =\> "xml" codec =\> multiline { pattern =\> "Request\>" negate =\> true what =\> "previous" } } } filter { xml { source =\> "message" xpath =\> [ "/Requests/Request/Service/text()", "service", "/Requests/Request/Version/text()", "version", "/Requests/Request/Operation/text()", "operation", "/Requests/Request/SubOperation/text()", "sub\_operation", "/Requests/Request/Resources/text()", "resources", "/Requests/Request/Path/text()", "path", "/Requests/Request/QueryString/text()", "query\_string", "/Requests/Request/Body/text()", "body", "/Requests/Request/HttpMethod/text()", "http\_method", "/Requests/Request/StartTime/text()", "start\_time", "/Requests/Request/EndTime/text()", "end\_time", "/Requests/Request/TotalTime/text()", "total\_time", "/Requests/Request/RemoteAddr/text()", "remote\_addr", "/Requests/Request/RemoteHost/text()", "remote\_host", "/Requests/Request/Host/text()", "host", "/Requests/Request/RemoteUser/text()", "remote\_host", "/Requests/Request/ResponseStatus/text()", "response\_status", "/Requests/Request/ResponseLength/text()", "response\_length", "/Requests/Request/ResponseContentType/text()", "response\_content\_type", "/Requests/Request/CacheResult/text()", "cache\_result", "/Requests/Request/MissReason/text()", "miss\_reason", "/Requests/Request/Failed/text()", "failed" ] target =\> "xml\_value" } #mutate { #remove\_field =\> [tags, host, message, xml\_value] #} } output { stdout {} } XML contents are: >?xml version="1.0" encoding="UTF-8" ?\> >Requests\>>Request id="34"\> >Service\>>/Service\> >Version\>>/Version\> >Operation\>>/Operation\> >SubOperation\>>/SubOperation\> >Resources\>>/Resources\> >Path\>null>/Path\> >QueryString\>>/QueryString\> >Body\> >/Body\> >HttpMethod\>GET>/HttpMethod\> >StartTime\>2019-10-30T22:50:39.107Z>/StartTime\> >EndTime\>2019-10-30T22:50:43.55Z>/EndTime\> >TotalTime\>4,443>/TotalTime\> >RemoteAddr\>10.2.142.1>/RemoteAddr\> >RemoteHost\>10.2.142.1>/RemoteHost\> >Host\>[dev-alta-geo04.altalisdev.com](http://dev-alta-geo04.altalisdev.com)>/Host\> >RemoteUser\>anonymous>/RemoteUser\> >ResponseStatus\>200>/ResponseStatus\> >ResponseLength\>0>/ResponseLength\> >ResponseContentType\>>/ResponseContentType\> >CacheResult\>>/CacheResult\> >MissReason\>>/MissReason\> >Failed\>false>/Failed\> >/Request\> >Request id="35"\> >Service\>>/Service\> >Version\>>/Version\> >Operation\>>/Operation\> >SubOperation\>>/SubOperation\> >Resources\>>/Resources\> >Path\>/index.html>/Path\> >QueryString\>>/QueryString\> >Body\> >/Body\> >HttpMethod\>GET>/HttpMethod\> >StartTime\>2019-10-30T22:50:43.581Z>/StartTime\> >EndTime\>2019-10-30T22:50:43.612Z>/EndTime\> >TotalTime\>31>/TotalTime\> >RemoteAddr\>10.2.142.1>/RemoteAddr\> >RemoteHost\>10.2.142.1>/RemoteHost\> >Host\>[dev-alta-geo04.altalisdev.com](http://dev-alta-geo04.altalisdev.com)>/Host\> >RemoteUser\>anonymous>/RemoteUser\> >ResponseStatus\>200>/ResponseStatus\> >ResponseLength\>697>/ResponseLength\> >ResponseContentType\>text/html;charset=UTF-8>/ResponseContentType\> >CacheResult\>>/CacheResult\> >MissReason\>>/MissReason\> >Failed\>false>/Failed\> >/Request\> >/Requests\> ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2019, 6:15pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/2 "2019-11-05T18:15:30Z")
Please edit your post and use [markdown](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet) so that the XML is readable. ---
**Author:** ![kcb](https://avatars.discourse-cdn.com/v4/letter/k/da6949/32.png) [@kcb](https://discuss.elastic.co/u/kcb)\ **Post date:** [November 5, 2019, 8:02pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/4 "2019-11-05T20:02:28Z")
Post has been edited to make XML more readable ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 5, 2019, 9:24pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/5 "2019-11-05T21:24:28Z")
The problem is your multiline codec. It is not combining lines into valid XML. The first messages are ``` \n...false \n...false ``` When using target the xml filter will actually parse the first one, but the xpath option is completely unforgiving of any junk around the XML. Change your codec to use ``` pattern => " true what => "previous" ``` Then you need some filters to clean up the results ``` if [message] =~ /<\?xml/ { drop {} } mutate { gsub => ["message", "<(/)?Requests>", ""] } ``` That leaves you with valid XML. You have some typos in your xpath expressions you will need to clean. I actually would not do it using xpath. I don't like that everything is an array. Instead I would use ``` target => "xml_value" store_xml => true force_array => false ``` Now I realize you want to everything to have different names, but to get the names you want you basically add an underscore before an uppercase letter (except at start of key name) then fold to lowercase. That is easy enough to do in ruby... ``` ruby { code => ' x = event.get("xml_value") if x x.each { |k, v| newk = k.gsub(/(?!^)([A-Z])/, "_\\1") newk = newk.downcase event.remove("[xml_value][#{k}]") event.set("[xml_value][#{newk}]", v) } end ' } ``` If you are new to Ruby regexps, (?!^) is a negative lookahead assertion which means do not match when ^ is at the start of the pattern. So this pattern only matches inside a key value, not at its start. ---
**Author:** ![kcb](https://avatars.discourse-cdn.com/v4/letter/k/da6949/32.png) [@kcb](https://discuss.elastic.co/u/kcb)\ **Post date:** [November 6, 2019, 4:53pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/6 "2019-11-06T16:53:25Z")
We have implemented the changes as outlined and we are now getting error: :exception=\>#\ "D:/logstash-7.4.1/conf/geoserver\_audit\_20191030\_2.log" sincedb\_path =\> "nul" start\_position =\> "beginning" type =\> "xml" codec =\> multiline { pattern =\> "\ true what =\> "previous" } } } filter { xml { source =\> "message" target =\> "xml\_value" store\_xml =\> true force\_array =\> false } if [message] =~ /\ ["message", "\<(/)?Requests\>", ""] } ruby { code =\> ' x = event.get("xml\_value") if x x.each { |k, v| newk = k.gsub(/(?!^)([A-Z])/, "\_\1") newk = newk.downcase event.remove("[xml\_value][#{k}]") event.set("[xml\_value][#{newk}]", v) } end ' } } output { stdout {} } The full error message is: [2019-11-06T09:43:54,577][WARN][logstash.filters.xml][main] Error parsing xml with XmlSimple {:source=\>"message", :value=\>"\", :exception=\>#\, :backtrace=>["D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:720:in `get\_attributes'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:464:in `collapse'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:194:in `xml\_in'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.0.7/lib/logstash/filters/xml.rb:185:in `filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi\_filter'", "org/jruby/RubyArray.java:1800:in `each'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:159:in `multi\_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi_filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/java_pipeline.rb:243:in `block in start\_workers'"]} [2019-11-06T09:43:54,718][WARN][logstash.filters.xml][main] Error parsing xml with XmlSimple {:source=\>"message", :value=\>"\\n \n \n \n \n \n null\n \n \n \n \n GET\n 2019-10-30T22:50:39.107Z\n 2019-10-30T22:50:43.55Z\n 4,443\n 10.2.142.1\n 10.2.142.1\n [dev-alta-geo04.altalisdev.com](http://dev-alta-geo04.altalisdev.com)\n anonymous\n 200\n 0\n \n \n \n false\n", :exception=\>#\ , :backtrace=\>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/treeparser.rb:28:in `parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:288:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:45:in `initialize'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in `xml_in'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml\_in'", "D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/logstash-filter-xml-4.0.7/lib/logstash/filters/xml.rb:185:in `filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:143:in `do\_filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1800:in `each'", "D:/logstash-7.4.1/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi\_filter'", "D:/logstash-7.4.1/logstash-core/lib/logstash/java\_pipeline.rb:243:in `block in start\_workers'"]} > D:/logstash-7.4.1/vendor/bundle/jruby/2.5.0/gems/awesome\_print-1.7.0/lib/awesome\_print/formatters/base\_formatter.rb:31: warning: constant ::Fixnum is deprecated ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 6, 2019, 5:02pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/7 "2019-11-06T17:02:22Z")
> [@kcb](#): > > if [message] =~ /\ mutate { gsub =\> ["message", "\<(/)?Requests\>", ""] } These have to come before the xml filter because the are transforming the message into valid xml. ---
**Author:** ![kcb](https://avatars.discourse-cdn.com/v4/letter/k/da6949/32.png) [@kcb](https://discuss.elastic.co/u/kcb)\ **Post date:** [November 6, 2019, 5:18pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/8 "2019-11-06T17:18:03Z")
Thanks, we no longer get XML errors. However the console display is not quite what we were expecting. Our XML contains 2 records, one for id 34 and one for id 35. When we initially run logstash, it displays all the XML values for id 34, but not for id 35. When I hit Ctrl-C to abort the script, it then displayed the XML values for id 35. Is this expected behavior? Is there a way to get the console display to immediately dispaly the XML values for both rows? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [November 6, 2019, 5:57pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/9 "2019-11-06T17:57:40Z")
Use the [auto\_flush\_interval](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html#plugins-codecs-multiline-auto_flush_interval) on the multiline codec. ---
**Author:** ![kcb](https://avatars.discourse-cdn.com/v4/letter/k/da6949/32.png) [@kcb](https://discuss.elastic.co/u/kcb)\ **Post date:** [November 6, 2019, 6:24pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/10 "2019-11-06T18:24:18Z")
Thanks. Console output is now as expected. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 4, 2019, 6:24pm UTC](https://discuss.elastic.co/t/console-xml-works-but-not-file-xml/206645/11 "2019-12-04T18:24:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.