# XML Dynamic Parsing **URL:** \ **Category:** Logstash\ **Created:** [April 5, 2018, 3:05pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957 "2018-04-05T15:05:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\ **Post date:** [April 5, 2018, 3:05pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/1 "2018-04-05T15:05:55Z")
Still new to ELK. I'd like to parse an XML with Logstash and output to Elasticsearch. XML: > ``` > > O > false > . > . > . > > > mycontent1 > mycontent2 > . > . > . > > > mycontent1 > mycontent2 > . > . > . > > . > . > . > > > ``` My Config File: > ``` > input{ > file{ > path => "/usr/share/logstash/bin/myXML.xml" > start_position => beginning > } > } > #filter{ > # I DON'T KNOW WHAT TO PUT HERE > #} > output{ > elasticsearch{ > hosts => ["localhost:9200"] > user => elastic > password => changeme > } > stdout{} > } > > ``` I am confused as to how I should make my config file with the filters and what not. The XML file will have nearly hundreds of fields (hence the ". . . " and some have sub-fields (sort of a Object Oriented way of encapsulating the data within other data like a Class in Java). Is there a way to dynamically parse the XML file so I don't have to manually define the fields and the contents of them? Also, am I outputting to Elasticsearch correctly? Any help would be greatly appreciated 🙂 ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 5, 2018, 5:33pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/2 "2018-04-05T17:33:46Z")
Use ruby code plugin with Nokogiri ruby extension. ---
**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\ **Post date:** [April 5, 2018, 6:04pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/3 "2018-04-05T18:04:34Z")
@Suman_Reddy1 is there any documentation and examples of ruby and Nokogiri anywhere? I found this: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 5, 2018, 6:23pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/4 "2018-04-05T18:23:00Z")
You should use a multiline codec on the input to consume the entire file as a single event. There are many threads about how to do that. Then you can use a logstash xml filter to parse the XML ```auto filter { xml { source => "message" target => "theXML" } } ``` The output looks like this ```auto "theXML" => { "data3" => [ [0] { "REPEATINGTYPE" => "PageGroup", "rowdata" => [ [0] { "datax1" => [ [0] "mycontent1" ], "datax2" => [ [0] "mycontent2" ], "REPEATINGINDEX" => "subdata1" }, [1] { "datax1" => [ [0] "mycontent1" ], "datax2" => [ [0] "mycontent2" ], "REPEATINGINDEX" => "subdata2" } ] } ], "data1" => [ [0] "O" ], "data2" => [ [0] "false" ] } ``` or, if you set force\_array =\> false ```auto "theXML" => { "data3" => { "REPEATINGTYPE" => "PageGroup", "rowdata" => [ [0] { "datax2" => "mycontent2", "REPEATINGINDEX" => "subdata1", "datax1" => "mycontent1" }, [1] { "datax2" => "mycontent2", "REPEATINGINDEX" => "subdata2", "datax1" => "mycontent1" } ] }, "data1" => "O", "data2" => "false" } ``` ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 5, 2018, 6:26pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/5 "2018-04-05T18:26:15Z")
Ok. My bad, I didnt see your configuration. With your configuration, logstash will read each line as a new event. To fix that use multiline codec. Multiline codec will aggregate multiple lines into a single log event, in this case it will create one xml file. There are plenty of sample around multiline please go through them. As I am on mobile device unable to give you exact config. Secondly, once we aggregated and created a single xml file use ruby code. ruby{ code =\> “ require nokogiri xml=event.get(“message”) \<\> “ } > **[Using The Nokogiri Gem To Parse Nested XML Data In Ruby (Example)](https://coderwall.com/p/03r98q/using-the-nokogiri-gem-to-parse-nested-xml-data-in-ruby)** > > A protip by 123aswin123 about ruby, rails, ror, xml, and nokogiri. Above is a reference how to parse xml ---
**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\ **Post date:** [April 5, 2018, 7:20pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/6 "2018-04-05T19:20:12Z")
@Badger does the Logstash XML filter work to dynamically parse out every tag in the XML? In other words, do I have to specify each field in the filter that exists in my XML? ---
**Author:** ![AloysiusParedes](https://avatars.discourse-cdn.com/v4/letter/a/e9c0ed/32.png) [@AloysiusParedes](https://discuss.elastic.co/u/AloysiusParedes)\ **Post date:** [April 5, 2018, 7:21pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/7 "2018-04-05T19:21:48Z")
@Suman_Reddy1 Thank you. I will take a look into the Ruby and Nokogiri. If you have any more info once you have time, I'd really appreciate that too. I'll try and keep learning. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 5, 2018, 8:00pm UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/8 "2018-04-05T20:00:50Z")
> [@AloysiusParedes](#): > > does the Logstash XML filter work to dynamically parse out every tag in the XML? Yes. The filter I showed parsed out all the fields in the XML ---
**Author:** ![Suman\_Reddy1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/suman_reddy1/32/27303_2.png) [@Suman\_Reddy1](https://discuss.elastic.co/u/Suman_Reddy1)\ **Post date:** [April 6, 2018, 8:24am UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/9 "2018-04-06T08:24:51Z")
Below is a recursive way of iterating all elements in an xml ``` ruby { code => " require 'nokogiri'def iterative(ele) ele.children.each do |tempNode| if tempNode.text? puts tempNode.content else iterative(tempNode) end end end xml_doc = Nokogiri::XML.parse(event.get('xml-data')) iterative(xml_doc)" } ``` Above is the sample, which we used to parse xml do some inline masking on the data. This should give you some insight on xml processing. If you dont have to do much manipulation on XML, I would suggest Badger solution rather than this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 4, 2018, 8:24am UTC](https://discuss.elastic.co/t/xml-dynamic-parsing/126957/10 "2018-05-04T08:24:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.