# XML on Elasticsearch

**URL:** https://discuss.elastic.co/t/xml-on-elasticsearch/142601
**Category:** Logstash
**Created:** [August 1, 2018, 2:50pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601 "2018-08-01T14:50:52Z")
**Posts on this page:** 11
**Page:** 1

<div class="post-metadata">

### Author: ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [August 1, 2018, 2:50pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/1 "2018-08-01T14:50:53Z")

</div>

Hi Guys!

I need index file XML on cluster Elasticsearch, following a flux:

S3 -\> Logstash -\> Elasticsearch

I read about xpath on xml filter, BUT my xml is very large then I don't get map all xml

I need that each field on XML be a field on Elasticsearch, like this:

This is a very simple example a XML file:

`<?xml version="1.0" encoding="ISO-8859-1"?><FAT><DATA><CLIENT Name="bla bla bla" A_C="bla01" Id="001" CP="00981726"></CLIENT></DATA></FAT></xml>`

XML filter is a better option to do this ?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 3:14pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/2 "2018-08-01T15:14:57Z")

</div>

That is not valid XML (it opens \<CLIENT\> and closes \</CLIENTE\>). Also, you need to strip off the \</xml\> which can be done using

```
mutate { gsub => ["message", "</xml>$", ""] }

```

Then you can parse it using

```
xml { source => "message" store_xml => true target => "theXML" force_array => false }

```

which gets you

```
    "theXML" => {
    "DATA" => {
        "CLIENT" => {
             "A_C" => "bla01",
              "CP" => "00981726",
            "Name" => "bla bla bla",
              "Id" => "001"
        }
    }
}
```

---

<div class="post-metadata">

### Author: ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [August 1, 2018, 4:29pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/3 "2018-08-01T16:29:09Z")

</div>

Hi @Badger

I configured mutate and xml, but I'm receveid this error:

`:exception=>#<REXML::ParseException: missing attribute quote Line: 1 Position: 62576 Last 80 unconsumed characters:`

My config:

`filter { mutate { gsub => ["message", "</xml>$", ""] } }`

`filter { xml { source => "message" store_xml => true target => "theXML" force_array => false } }`

The index is created on Elasticsearch, but all fields without field `message`

Like this:

`"_index" : "teste-2018.08", "_type" : "doc", "_id" : "_1FW9mQBCtVZHh-PRMtz", "_score" : 1.0, "_source" : { "tags" : ["_xmlparsefailure"], "@timestamp" : "2018-08-01T16:33:22.758Z", "@version" : "1", "message" : "<?xml version=\\\"1.0\\\" encoding=\\\"ISO-8859-1\\\"?><FAT ...`

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 4:42pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/4 "2018-08-01T16:42:11Z")

</div>

Immediately after that error message it will show the XML that has an issue. I suspect your XML looks like this

```
<foo><bar a=1/></foo>

```

That is not valid XML. It has to be

```
<foo><bar a="1"/></foo>

```

You might be able to fix the "XML" using stuff like

```
mutate { gsub => ["message", "( a=)([^/>]+)([/>])", '\1"\2"\3' ] }
```

---

<div class="post-metadata">

### Author: ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [August 1, 2018, 4:53pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/5 "2018-08-01T16:53:28Z")

</div>

@Badger

Yes, my XML is valid, look:

`<?xml version=\\\"1.0\\\" encoding=\\\"ISO-8859-1\\\"?><FAT><DATA><CLIENT Nome=\\\"bla bla\\\" A_C=\\\"bla01 - .\\\" Id=\\\"0010\\\" CP=\\\"00098281\\\"></CLIENT></DATA></FAT></xml>`

This is only a part of XML, there is much that 2.000 lines

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 4:57pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/6 "2018-08-01T16:57:00Z")

</div>

As I said, immediately after the error message is the problematic XML.

```
[2018-08-01T12:55:15,376][WARN][logstash.filters.xml] Error parsing xml with XmlSimple {:source=>"message", :value=>"<foo><bar a=1></foo>", :exception=>#<REXML::ParseException: missing attribute quote
Line: 1
Position: 20
Last 80 unconsumed characters:
<bar a=1></foo>>, 

```

Are you able to post the full error message including the unconsumed characters?

---

<div class="post-metadata">

### Author: ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [August 1, 2018, 5:05pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/7 "2018-08-01T17:05:53Z")

</div>

@Badger Sure!

`:exception=>#<REXML::ParseException: missing attribute quote Line: 1 Position: 102125 Last 80 unconsumed characters: <CLIENT Nome=\"bla bla \" A_C=\"bla01 - .\" Id>, :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/baseparser.rb:374:in`pull\_event'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/baseparser.rb:185:in `pull'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/parsers/treeparser.rb:23:in`parse'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:288:in `build'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/rexml/document.rb:45:in`initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:971:in `parse'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:164:in`xml\_in'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/xml-simple-1.1.5/lib/xmlsimple.rb:203:in `xml_in'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-xml-4.0.5/lib/logstash/filters/xml.rb:182:in`filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in`block in multi\_filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in`multi\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter\_delegator.rb:47:in `multi_filter'", "(eval):69:in`block in filter\_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:445:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:424:in`worker\_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:386:in `block in start_workers'"]}`

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 5:09pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/8 "2018-08-01T17:09:20Z")

</div>

The log message adds a \> to the XML. So the end of the XML it is consuming is

```
CLIENT Nome=\"bla bla \" A_C=\"bla01 - .\" Id

```

I think the XML might be truncated. What input are you using?

---

<div class="post-metadata">

### Author: ![Fram\_Souza](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fram_souza/32/24833_2.png) [@Fram\_Souza](https://discuss.elastic.co/u/Fram_Souza)
#### Post date: [August 1, 2018, 5:12pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/9 "2018-08-01T17:12:40Z")

</div>

@Badger

My input is very simple:

`input { s3 { "bucket" => "fat" "prefix" => "XML/XX/2018/07/13/1/00" } }`

Each file xml have at about 350 KB

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 1, 2018, 5:50pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/10 "2018-08-01T17:50:48Z")

</div>

I cannot reconcile that error message with the source code unless the input event literally ended at Id.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 29, 2018, 5:51pm UTC](https://discuss.elastic.co/t/xml-on-elasticsearch/142601/11 "2018-08-29T17:51:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.