# How to properly read and ingest XML file into Elastic

**URL:** <https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582>\
**Category:** Logstash\
**Created:** [May 30, 2019, 5:17pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582 "2019-05-30T17:17:45Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 30, 2019, 5:17pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/1 "2019-05-30T17:17:45Z")

</div>

Hello,

I have been trying to ingest an XML file into Elastic using Logstash. I have been using Xpath, target, source in the XML filter options and gsub for mutate filter, however, I keep running into an Error parsing xml with XmlSimple....ParseException: Missing end tag for ' '. How does one properly format this config file or ingest an XML file?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 30, 2019, 7:47pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/2 "2019-05-30T19:47:06Z")

</div>

Can you post your XML?

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 2:30pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/4 "2019-05-31T14:30:14Z")

</div>

Sure here is the XML. In between the there are more with the exact same fields as the ones above. Different values of course but with pretty much the exact same format. Exact same number of fields. (View Down Below)

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 31, 2019, 2:31pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/5 "2019-05-31T14:31:21Z")

</div>

Please do not post pictures of text, just post the text.

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 2:33pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/7 "2019-05-31T14:33:50Z")

</div>

```
> <?xml version="1.0" encoding="utf-8"?>
<PlanRequests>
   <Header export_date="25-Jun-2018 18:00" query="test_query extents"/>
   <ChgTestRev test_id="TD-TD-00000004" test_rev="001" type="TT-TD" status="TEst" url="https://test.some.com/test/#com.company.more.testfx.test.write.showObject;nid=tgERGEsdEWR" last_modified_date="21-Mar-2017 12:07">
      <Property name="test_data">TestData, Test Data</Property>
      <Property name="test_data2">TestData, Test Data/002</Property>
      <Property name="TDTest">TestData</Property>
      <Property name="TDTest2">TestData</Property>
      <Property name="TDTEst3">TestData</Property>
      <Property name="TDTest4"></Property>
      <Property name="test_data3"></Property>
      <Property name="test_data4">TestData</Property>
      <Property name="test_data5">TestData, TestData (testt)</Property>
      <Property name="test_data6">TestData, TestData (twett)</Property>
      <Property name="test_data7">TestData, TestData (ttett)</Property>
      <Property name="test_data8"></Property>
      <Property name="test_data9">TestData/TestData</Property>
      <Property name="test_data10"><![CDATA[<p>testingtest test test</p>

<p>Make sure <em><strong><span style="background-color:rgb(64, 224, 208)">spell check </span></strong></em>works</p>]]></Property>
      <Property name="test_data57"><![CDATA[<p>test test <span style="color:rgb(128, 0, 0)">make sure </span>spell check works, bolding et.&nbsp; -- ty</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>&nbsp;</p>
<h2 style="font-style: italic;">test again for <u><strong>allot of words </strong></u>and test</h2>]]></Property>
      <Property name="test_data11">TestData TestData</Property>
      <Property name="test_data12"></Property>
      <Property name="test_data13"></Property>
      <Property name="test_data14"></Property>
      <Property name="test_data15">TestData</Property>
      <Property name="test_data16">test.data@testing.com</Property>
      <Property name="test_data17">Test A. Testing</Property>
      <Property name="test_data18">Test test, test, test</Property>
      <Property name="test_data19"></Property>
      <Property name="test_data20">Test Data</Property>
      <Property name="test_data21">gsdgge</Property>
      <Property name="test_data22"></Property>
      <Property name="test_data23">TestData</Property>
      <Property name="test_data24"></Property>
      <Property name="test_data25">test, test, test, test</Property>
      <Property name="test_data26">test, test, test, test</Property>
      <Property name="test_data27"></Property>
      <Property name="test_data28">2003</Property>
      <Property name="test_data29">2005</Property>
      <Property name="test_data30"></Property>
      <Property name="test_data31"></Property>
      <Property name="test_data32">TestData, TestData (testt)</Property>
      <Property name="test_data33"></Property>
      <Property name="test_data34"></Property>
      <Property name="test_data35"></Property>
      <Property name="test_data36">TestData, TestData (testt)</Property>
      <Property name="test_data37"></Property>
      <Property name="test_data38">TEstDATa200</Property>
      <Property name="test_data39">test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work: test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work</Property>
      <Property name="test_data40"></Property>
      <Property name="test_data41"></Property>
      <Property name="test_data42"></Property>
      <Property name="test_data43"></Property>
      <Property name="test_data44"></Property>
      <Property name="test_data45"><![CDATA[25.25 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 25.25 - Test Data]]></Property>
      <Property name="test_data46"></Property>
      <Property name="test_data47"></Property>
      <TestImpact test_data_48="23-Mar-2017 12:07">
         <Program test_data49="TestData" test_data50="TestData" test_data51="1998" test_data52="1994" test_data53="Yes" test_data54="TDE" test_data55="Red" test_data56=""/>
      </TestImpact>
   </ChgTestRev>
</PlanRequests>
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 31, 2019, 2:39pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/8 "2019-05-31T14:39:12Z")

</div>

Even though it has an extra "\> " at the beginning, I have no problem parsing that using an xml filter.

```
xml { source => "message" target => "theXML" }
```

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 2:42pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/9 "2019-05-31T14:42:25Z")

</div>

Is that regardless of any other xml filter options?

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 2:50pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/10 "2019-05-31T14:50:03Z")

</div>

I'm getting this error, it's not creating the index:

[2019-05-31T10:47:20,937][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=\>500, :url=\>"[http://localhost:9200/\_xpack/monitoring/\_bulk?system\_id=logstash&system\_api\_version=2&interval=1s](http://localhost:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s)"}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 31, 2019, 2:59pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/11 "2019-05-31T14:59:17Z")

</div>

If logstash is seeing a 500 error from elasticsearch I would expect there to be a more informative error message in the elasticsearch logs.

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 3:20pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/12 "2019-05-31T15:20:18Z")

</div>

[2019-05-31T10:49:44,085][WARN][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document  
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]  
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]  
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:1.8.0\_201]  
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0\_201]  
at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:1.8.0\_201]  
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?]  
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) ~[?:?]  
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:570) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:820) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:808) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:706) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:170) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:98) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:128) ~[?:?]  
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:349) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:461) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:174) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:513) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.ingest.PipelineExecutionService$1.doRun(PipelineExecutionService.java:87) [elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]  
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0\_201]  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0\_201]  
at java.lang.Thread.run(Unknown Source) [?:1.8.0\_201]  
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];  
at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:183) ~[elasticsearch-6.3.2.jar:6.3.2]  
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:806) ~[elasticsearch-6.3.2.jar:6.3.2]  
... 20 more

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [May 31, 2019, 3:25pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/13 "2019-05-31T15:25:53Z")

</div>

What's the rest of your pipeline look like. The error says forbidden, im thinking your output may be incorrect.

---

<div class="post-metadata">

**Author:** ![edster](https://avatars.discourse-cdn.com/v4/letter/e/da6949/32.png) [@edster](https://discuss.elastic.co/u/edster)\
**Post date:** [May 31, 2019, 3:34pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/14 "2019-05-31T15:34:13Z")

</div>

I'm using the same output I've used for other indices it's the same. Doesn't look any different.

#file:  
input {  
file {  
path =\> "path to xml"  
}  
}

filter {  
xml {  
source =\> "message"  
target =\> "theXML"  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "chgtest"  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 31, 2019, 3:41pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/15 "2019-05-31T15:41:41Z")

</div>

> [@edster](#):
>
> ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

Looks like you ran out of disk space (ES sets indexes to read-only when disk utilization hits 95%). You need to free up some space and [reset](https://www.elastic.co/guide/en/elasticsearch/reference/current/disk-allocator.html) the read-only flag.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2019, 3:41pm UTC](https://discuss.elastic.co/t/how-to-properly-read-and-ingest-xml-file-into-elastic/183582/16 "2019-06-28T15:41:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.