# How to handle XML file "Last ## unconsumed characters" error

**URL:** https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385
**Category:** Logstash
**Created:** [July 1, 2019, 6:27pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385 "2019-07-01T18:27:09Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 1, 2019, 6:27pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/1 "2019-07-01T18:27:09Z")

</div>

Elastic has an issue parsing xml. Returns error "Error parsing xml with XmlSimple...." Also "Last ## unconsumed characters". Logstash config file is as follows

> input {  
> file {  
> path =\> "filepath"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> }  
> }  
> filter {  
> xml {  
> source =\> "message"  
> target =\> "theXML"  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> ["hostserver:9200"]  
> index =\> "crtest"  
> }  
> }

---

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 1, 2019, 6:54pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/2 "2019-07-01T18:54:17Z")

</div>

```
<?xml version="1.0" encoding="utf-8"?>
<PlanRequests>
   <Header export_date="25-Jun-2018 18:00" query="test_query extents"/>
   <ChgTestRev test_id="TD-TD-00000004" test_rev="001" type="TT-TD" status="TEst" url="https://test.some.com/test/#com.company.more.testfx.test.write.showObject;nid=tgERGEsdEWR" last_modified_date="21-Mar-2017 12:07">
      <Property name="test_data">TestData, Test Data</Property>
      <Property name="test_data2">TestData, Test Data/002</Property>
      <Property name="TDTest">TestData</Property>
      <Property name="TDTest2">TestData</Property>
      <Property name="TDTEst3">TestData</Property>
      <Property name="TDTest4"></Property>
      <Property name="test_data3"></Property>
      <Property name="test_data4">TestData</Property>
      <Property name="test_data5">TestData, TestData (testt)</Property>
      <Property name="test_data6">TestData, TestData (twett)</Property>
      <Property name="test_data7">TestData, TestData (ttett)</Property>
      <Property name="test_data8"></Property>
      <Property name="test_data9">TestData/TestData</Property>
      <Property name="test_data10"><![CDATA[<p>testingtest test test</p>

<p>Make sure <em><strong><span style="background-color:rgb(64, 224, 208)">spell check </span></strong></em>works</p>]]></Property>
      <Property name="test_data57"><![CDATA[<p>test test <span style="color:rgb(128, 0, 0)">make sure </span>spell check works, bolding et.&nbsp; -- ty</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<h2 style="font-style: italic;">test again for <u><strong>allot of words </strong></u>and test</h2>]]></Property>
      <Property name="test_data11">TestData TestData</Property>
      <Property name="test_data12"></Property>
      <Property name="test_data13"></Property>
      <Property name="test_data14"></Property>
      <Property name="test_data15">TestData</Property>
      <Property name="test_data16">test.data@testing.com</Property>
      <Property name="test_data17">Test A. Testing</Property>
      <Property name="test_data18">Test test, test, test</Property>
      <Property name="test_data19"></Property>
      <Property name="test_data20">Test Data</Property>
      <Property name="test_data21">gsdgge</Property>
      <Property name="test_data22"></Property>
      <Property name="test_data23">TestData</Property>
      <Property name="test_data24"></Property>
      <Property name="test_data25">test, test, test, test</Property>
      <Property name="test_data26">test, test, test, test</Property>
      <Property name="test_data27"></Property>
      <Property name="test_data28">2003</Property>
      <Property name="test_data29">2005</Property>
      <Property name="test_data30"></Property>
      <Property name="test_data31"></Property>
      <Property name="test_data32">TestData, TestData (testt)</Property>
      <Property name="test_data33"></Property>
      <Property name="test_data34"></Property>
      <Property name="test_data35"></Property>
      <Property name="test_data36">TestData, TestData (testt)</Property>
      <Property name="test_data37"></Property>
      <Property name="test_data38">TEstDATa200</Property>
      <Property name="test_data39">test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work: test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work</Property>
      <Property name="test_data40"></Property>
      <Property name="test_data41"></Property>
      <Property name="test_data42"></Property>
      <Property name="test_data43"></Property>
      <Property name="test_data44"></Property>
      <Property name="test_data45"><![CDATA[25.25 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 25.25 - Test Data]]></Property>
      <Property name="test_data46"></Property>
      <Property name="test_data47"></Property>
      <TestImpact test_data_48="23-Mar-2017 12:07">
         <Program test_data49="TestData" test_data50="TestData" test_data51="1998" test_data52="1994" test_data53="Yes" test_data54="TDE" test_data55="Red" test_data56=""/>
      </TestImpact>
   </ChgTestRev>
</PlanRequests>

```

Is an example of part of the xml file being ingested. The field names would be the "test\_data" or "TestData" fields and the values would be those which equal are in quotes and equal to it or are in between opening and closing tags.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 2, 2019, 8:08pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/3 "2019-07-02T20:08:23Z")

</div>

A file input in tail mode (the default) will consume a file one line at a time. You could change the file input to [read](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_read_mode) mode, making sure you understand the default [file\_completion\_action](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-file_completed_action).

Or else use a multiline codec with a pattern that never matches. For example

```
codec => multiline {
    pattern => "^Spalanzani"
    negate => true
    what => previous
    auto_flush_interval => 1
}
```

---

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 3, 2019, 3:54pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/4 "2019-07-03T15:54:04Z")

</div>

Hi @Badger,

Sorry I have a few questions. The codec options seemed to work, however, everything is being pulled in as one whole document but they should be pulled in as multiple objects (not for this file but for one with more objects like the one above). if that makes sense. Is there a way to split those or separate them?

Also, is it possible to not include the message field since it seems to pretty much repeat the whole doc over again? Would just be wasted space since its being pulled in those fields. The store\_xml option didn't work.

The Properties also come in as an array with name (field) and content (value) pairs. Is it possible just have these pulled in not as an array of pairs but just have the name as fields for the doc and their corresponding content (value)?

Does the pattern control which doc or part of the xml file is included or no? In other words is there a way to filter the xml document for objects that one does not want included based on a field?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 3, 2019, 5:04pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/5 "2019-07-03T17:04:26Z")

</div>

You really need to provide an example with your questions. Otherwise it is unclear exactly what you are asking.

When you say more than one object, an xml filter cannot parse an event like

```
<a>1</a><a>2</a>

```

It will fail with "attempted adding second root element to document". If you have XML like

```
<a><b>1</b><b>2</b></a>

```

The the xml filter will create an array which you can split.

```
    xml { source => "message" store_xml => true target => "theXML" remove_field => ["message"] }
    split { field => "[theXML][b]" }

```

results in

```
{
      "path" => "/home/user/foo.txt",
"@timestamp" => 2019-07-03T17:02:06.239Z,
  "@version" => "1",
    "theXML" => {
    "b" => "1"
}
}
{
      "path" => "/home/user/foo.txt",
"@timestamp" => 2019-07-03T17:02:06.239Z,
  "@version" => "1",
    "theXML" => {
    "b" => "2"
}
}

```

So if you ask a more specific question I will try to answer it.

---

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 9, 2019, 2:56pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/6 "2019-07-09T14:56:21Z")

</div>

So as an example:

```
 (1) <PlanRequests>
         ....<field1>
         ....<field2>
         ....<field3>
         ....<field4>
   </PlanRequests>
 (2) <PlanRequests>
         ....<field1>
         ....<field2>
         ....<field3>
         ....<field4>
   </PlanRequests>

```

I mean like splitting these up. Having (1) as one object and (2) as another and so on. They have the same fields but are separate objects. Im trying to get both or rather multiple of these as their docs within an index.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 9, 2019, 3:25pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/7 "2019-07-09T15:25:37Z")

</div>

If you have a file that looks like

```
<?xml version="1.0" encoding="utf-8"?>
<PlanRequests>
   <Header export_date="25-Jun-2018 18:00" query="test_query extents"/>
</PlanRequests>
<PlanRequests>
   <Header export_date="26-Jun-2018 18:00" query="foo"/>
</PlanRequests>

```

then use a multiline codec to read each object

```
codec => multiline { pattern => "</PlanRequests>" negate => true what => next auto_flush_interval => 1 }
```

---

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 12, 2019, 3:52pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/8 "2019-07-12T15:52:21Z")

</div>

Hi Badger,

It's working and pulling data way better than it before. Thank you. There is only one more issue which is that the beginning \<?xml version...\> and an extra line or two part of the xml but not for the separate objects creates errors and pulls in an extra object with the tag "xmlParseFailure". When these lines are removed the errors don't occur and all data pulled is valid. Having to remove these lines manually would be tedious since it would need to be done often. Is there anyway to remove these lines or ignore them when ingesting this or any xml file?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 12, 2019, 4:34pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/9 "2019-07-12T16:34:07Z")

</div>

The '\<?xml version...' does not cause a parsing problem in the example I gave. If you have extra text that does cause a problem then you could use mutate+gsub to remove it.

---

<div class="post-metadata">

### Author: ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)
#### Post date: [July 12, 2019, 6:44pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/10 "2019-07-12T18:44:38Z")

</div>

The lines include an opening tag with a different name at the beginning of the xml file and a closing tag at the end. And an extra header for it but its not part of an of the objects. The mutate+gsub take field names right? I wouldn't be able to remove those 4 lines with it no?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 12, 2019, 7:04pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/11 "2019-07-12T19:04:23Z")

</div>

I would expect mutate+gsub to be able to do it. For example, to remove the \<?xml version... you could use

```
    mutate { gsub => [ "message", "^<\?xml[^
]+
", "" ] }

```

Use literal newlines inside the pattern. So that is start of line, followed by \<?xml followed by one or more not-newline followed by newline.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 9, 2019, 7:04pm UTC](https://discuss.elastic.co/t/how-to-handle-xml-file-last-unconsumed-characters-error/188385/12 "2019-08-09T19:04:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.