Hello,
I have been trying to ingest an XML file into Elastic using Logstash. I have been using Xpath, target, source in the XML filter options and gsub for mutate filter, however, I keep running into an Error parsing xml with XmlSimple....ParseException: Missing end tag for ' '. How does one properly format this config file or ingest an XML file?
Can you post your XML?
Sure here is the XML. In between the there are more with the exact same fields as the ones above. Different values of course but with pretty much the exact same format. Exact same number of fields. (View Down Below)
Please do not post pictures of text, just post the text.
> <?xml version="1.0" encoding="utf-8"?>
<PlanRequests>
<Header export_date="25-Jun-2018 18:00" query="test_query extents"/>
<ChgTestRev test_id="TD-TD-00000004" test_rev="001" type="TT-TD" status="TEst" url="https://test.some.com/test/#com.company.more.testfx.test.write.showObject;nid=tgERGEsdEWR" last_modified_date="21-Mar-2017 12:07">
<Property name="test_data">TestData, Test Data</Property>
<Property name="test_data2">TestData, Test Data/002</Property>
<Property name="TDTest">TestData</Property>
<Property name="TDTest2">TestData</Property>
<Property name="TDTEst3">TestData</Property>
<Property name="TDTest4"></Property>
<Property name="test_data3"></Property>
<Property name="test_data4">TestData</Property>
<Property name="test_data5">TestData, TestData (testt)</Property>
<Property name="test_data6">TestData, TestData (twett)</Property>
<Property name="test_data7">TestData, TestData (ttett)</Property>
<Property name="test_data8"></Property>
<Property name="test_data9">TestData/TestData</Property>
<Property name="test_data10"><![CDATA[<p>testingtest test test</p>
<p>Make sure <em><strong><span style="background-color:rgb(64, 224, 208)">spell check </span></strong></em>works</p>]]></Property>
<Property name="test_data57"><![CDATA[<p>test test <span style="color:rgb(128, 0, 0)">make sure </span>spell check works, bolding et. -- ty</p>
<p> </p>
<p> </p>
<p> </p>
<h2 style="font-style: italic;">test again for <u><strong>allot of words </strong></u>and test</h2>]]></Property>
<Property name="test_data11">TestData TestData</Property>
<Property name="test_data12"></Property>
<Property name="test_data13"></Property>
<Property name="test_data14"></Property>
<Property name="test_data15">TestData</Property>
<Property name="test_data16">test.data@testing.com</Property>
<Property name="test_data17">Test A. Testing</Property>
<Property name="test_data18">Test test, test, test</Property>
<Property name="test_data19"></Property>
<Property name="test_data20">Test Data</Property>
<Property name="test_data21">gsdgge</Property>
<Property name="test_data22"></Property>
<Property name="test_data23">TestData</Property>
<Property name="test_data24"></Property>
<Property name="test_data25">test, test, test, test</Property>
<Property name="test_data26">test, test, test, test</Property>
<Property name="test_data27"></Property>
<Property name="test_data28">2003</Property>
<Property name="test_data29">2005</Property>
<Property name="test_data30"></Property>
<Property name="test_data31"></Property>
<Property name="test_data32">TestData, TestData (testt)</Property>
<Property name="test_data33"></Property>
<Property name="test_data34"></Property>
<Property name="test_data35"></Property>
<Property name="test_data36">TestData, TestData (testt)</Property>
<Property name="test_data37"></Property>
<Property name="test_data38">TEstDATa200</Property>
<Property name="test_data39">test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work: test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work:test what happens if make this very long does it wrapt appropriately - spel chek does not work</Property>
<Property name="test_data40"></Property>
<Property name="test_data41"></Property>
<Property name="test_data42"></Property>
<Property name="test_data43"></Property>
<Property name="test_data44"></Property>
<Property name="test_data45"><![CDATA[25.25 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 24.57 - Test Data, 25.25 - Test Data]]></Property>
<Property name="test_data46"></Property>
<Property name="test_data47"></Property>
<TestImpact test_data_48="23-Mar-2017 12:07">
<Program test_data49="TestData" test_data50="TestData" test_data51="1998" test_data52="1994" test_data53="Yes" test_data54="TDE" test_data55="Red" test_data56=""/>
</TestImpact>
</ChgTestRev>
</PlanRequests>Even though it has an extra "> " at the beginning, I have no problem parsing that using an xml filter.
xml { source => "message" target => "theXML" }Is that regardless of any other xml filter options?
I'm getting this error, it's not creating the index:
[2019-05-31T10:47:20,937][ERROR][logstash.outputs.elasticsearch] Encountered a retryable error. Will Retry with exponential backoff {:code=>500, :url=>"http://localhost:9200/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s"}
If logstash is seeing a 500 error from elasticsearch I would expect there to be a more informative error message in the elasticsearch logs.
[2019-05-31T10:49:44,085][WARN ][o.e.x.m.e.l.LocalExporter] unexpected error while indexing monitoring document
org.elasticsearch.xpack.monitoring.exporter.ExportException: ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$throwExportException$2(LocalBulk.java:128) ~[?:?]
at java.util.stream.ReferencePipeline$3$1.accept(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.ReferencePipeline$2$1.accept(Unknown Source) ~[?:1.8.0_201]
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.AbstractPipeline.copyInto(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.AbstractPipeline.evaluate(Unknown Source) ~[?:1.8.0_201]
at java.util.stream.ReferencePipeline.forEach(Unknown Source) ~[?:1.8.0_201]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.throwExportException(LocalBulk.java:129) ~[?:?]
at org.elasticsearch.xpack.monitoring.exporter.local.LocalBulk.lambda$doFlush$0(LocalBulk.java:111) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:43) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkRequestModifier.lambda$wrapActionListenerIfNeeded$0(TransportBulkAction.java:570) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.finishHim(TransportBulkAction.java:379) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$1.onFailure(TransportBulkAction.java:374) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:91) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.finishAsFailed(TransportReplicationAction.java:897) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockException(TransportReplicationAction.java:820) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:808) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.doRun(TransportReplicationAction.java:706) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:170) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction.doExecute(TransportReplicationAction.java:98) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:128) ~[?:?]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:349) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(TransportBulkAction.java:461) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction.doExecute(TransportBulkAction.java:174) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.bulk.TransportBulkAction.lambda$processBulkIndexIngestRequest$4(TransportBulkAction.java:513) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.ingest.PipelineExecutionService$1.doRun(PipelineExecutionService.java:87) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:725) [elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.2.jar:6.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_201]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_201]
at java.lang.Thread.run(Unknown Source) [?:1.8.0_201]
Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];
at org.elasticsearch.cluster.block.ClusterBlocks.indexBlockedException(ClusterBlocks.java:183) ~[elasticsearch-6.3.2.jar:6.3.2]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReroutePhase.handleBlockExceptions(TransportReplicationAction.java:806) ~[elasticsearch-6.3.2.jar:6.3.2]
... 20 more
What's the rest of your pipeline look like. The error says forbidden, im thinking your output may be incorrect.
I'm using the same output I've used for other indices it's the same. Doesn't look any different.
#file:
input {
file {
path => "path to xml"
}
}
filter {
xml {
source => "message"
target => "theXML"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "chgtest"
}
}
Looks like you ran out of disk space (ES sets indexes to read-only when disk utilization hits 95%). You need to free up some space and reset the read-only flag.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.