# Problems with parsing XML log files with XML filter of Logstash

**URL:** <https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796>\
**Category:** Logstash\
**Created:** [December 9, 2015, 8:35pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796 "2015-12-09T20:35:18Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Omer\_Uludag](https://avatars.discourse-cdn.com/v4/letter/o/b9e5f3/32.png) [@Omer\_Uludag](https://discuss.elastic.co/u/Omer_Uludag)\
**Post date:** [December 9, 2015, 8:35pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/1 "2015-12-09T20:35:18Z")

</div>

Hello together,

I tried multiple solutions for parsing my logs which are XML files to JSON in Logstash.  
One log file does look like in this way:

```
<log level="INFO" time="Wed Sep 09 09:18:48 EDT 2015" timel="1441804728245" id="123456789" cat="COMMUNICATION" comp="" host="127.0.0.0.1" req="" app="" usr="" thread="" origin="">
	<msg>
		<![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[3540], ntCoent-Length=[6660], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:426/CSI:-/Me:0/Total:426]]>
	</msg>
	<info>
	</info>
	<excp>
	</excp>
</log>
<log level="INFO" time="Wed Sep 09 09:18:48 EDT 2015" timel="1441804728245" id="123456789" cat="COMMUNICATION" comp="" host="127.0.0.0.1" req="" app="" usr="" thread="" origin="">
	<msg>
		<![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[3540], ntCoent-Length=[6660], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:426/CSI:-/Me:0/Total:426]]>
	</msg>
	<info>
	</info>
	<excp>
	</excp>
</log>
<log level="INFO" time="Wed Sep 09 09:18:48 EDT 2015" timel="1441804728245" id="123456789" cat="COMMUNICATION" comp="" host="127.0.0.0.1" req="" app="" usr="" thread="" origin="">
	<msg>
		<![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[3540], ntCoent-Length=[6660], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:426/CSI:-/Me:0/Total:426]]>
	</msg>
	<info>
	</info>
	<excp>
	</excp>
</log>

```

I have in a log file multiple logs (in this example 3).  
I tried this filter:

```
input {
file {
  path => "/path/to/file.log.*"
  start_position => "beginning"

}
}
filter{ multiline {
  pattern => "<log"
  negate => "true"
  what => "previous"
}
  xml {
  	store_xml => "false"
  	source => "message"
  xpath => [
     "/log/at.level", "level",
     "/log/at.time", "time",
     "/log/at.timel", "timel",
     "/log/at.id", "id",
     "/log/at.cat", "cat",
     "/log/at.comp", "comp",
     "/log/at.host", "host",
     "/log/at.req", "req",
     "/log/at.app", "app",
     "/log/at.usr", "usr",
     "/log/at.thread", "thread",
     "/log/at.origin", "origin",
     "/log/msg/text()","msg_txt"
  ]
  }

}
output {
  elasticsearch {
hosts => "localhost:9200"

}
}

```

Of course, the "at's", must be replaced with the at sign.  
But when I starting to run Logstash it creates really weird output. However, as a consequence Elasticsearch cannot read it.  
Maybe do you have some suggestions, where I miss something?

Best regards,  
Oemer

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 9, 2015, 9:08pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/2 "2015-12-09T21:08:52Z")

</div>

> But when I starting to run Logstash it creates really weird output.

Could you be more specific? To start with, is the multiline filter producing correctly joined lines?

---

<div class="post-metadata">

**Author:** ![Omer\_Uludag](https://avatars.discourse-cdn.com/v4/letter/o/b9e5f3/32.png) [@Omer\_Uludag](https://discuss.elastic.co/u/Omer_Uludag)\
**Post date:** [December 9, 2015, 9:33pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/3 "2015-12-09T21:33:45Z")

</div>

Hello Magnus,  
thank you for your reply.

The Output looks like in this way (excerpt)

```
Defaulting filter worker threads to 1 because there are some filters that might not work with multiple worker threads {:count_was=>4, :filters=>["multiline"], :level=>:warn}
Logstash startup completed
2015-12-09T21:11:24.687Z host T, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[4218], ntCoent-Length=[7592], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:453/CSI:-/Me:1/Total:454]]></msg><info></info><excp></excp></log>
2015-12-09T21:12:54.906Z host <log level="INFO" time="Tue Sep 08 17:41:27 EDT 2015" timel="1441748487311" id="123456789" cat="COMMUNICATION" comp="CNGW (WEB)" host="Test" req="" app="" usr="" thread="" origin=""><msg><![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[10270], ntCoent-Length=[19922], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:644/CSI:-/Me:0/Total:644]]></msg><info></info><excp></excp></log>
2015-12-09T21:12:54.908Z host TestTest<log level="INFO" time="Tue Sep 08 17:41:27 EDT 2015" timel="1441748487460" id="123456789" cat="COMMUNICATION" comp="CNGW (WEB)" host="Test" req="" app="" usr="" thread="" origin=""><msg><![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[4424], ntCoent-Length=[7944], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:561/CSI:-/Me:0/Total:561]]></msg><info></info><excp></excp></log>

```

After a certain time the output looks like this:  
TestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTestTest

Maybe my understanding is not correct.  
I use multiline because in my file, I have multiple log tags. Means for each log, one event should be created or (in rel db. one row)

Best regards

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 10, 2015, 6:45am UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/4 "2015-12-10T06:45:28Z")

</div>

What does "Test" come from? That string doesn't appear in your configuration AFAICT.

---

<div class="post-metadata">

**Author:** ![Omer\_Uludag](https://avatars.discourse-cdn.com/v4/letter/o/b9e5f3/32.png) [@Omer\_Uludag](https://discuss.elastic.co/u/Omer_Uludag)\
**Post date:** [December 10, 2015, 12:41pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/5 "2015-12-10T12:41:19Z")

</div>

Test comes from here:  
`<![CDATA[Method=GET URL=http://test.de/24dsdf3=0TReq(provider=Test, Decoding_Feat=[], Accept-Encoding=gzip, Accept=*/*) Result(Content-Encoding=[gzip], Content-Length=[3540], ntCoent-Length=[6660], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:426/CSI:-/Me:0/Total:426]]>`  
It is the first attribute in the URL: "provider"

---

<div class="post-metadata">

**Author:** ![Omer\_Uludag](https://avatars.discourse-cdn.com/v4/letter/o/b9e5f3/32.png) [@Omer\_Uludag](https://discuss.elastic.co/u/Omer_Uludag)\
**Post date:** [December 10, 2015, 6:09pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/6 "2015-12-10T18:09:18Z")

</div>

Test comes from the Provider attribute within the URL

---

<div class="post-metadata">

**Author:** ![Omer\_Uludag](https://avatars.discourse-cdn.com/v4/letter/o/b9e5f3/32.png) [@Omer\_Uludag](https://discuss.elastic.co/u/Omer_Uludag)\
**Post date:** [December 10, 2015, 10:33pm UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/7 "2015-12-10T22:33:48Z")

</div>

The question is also, if I need to specify a type or have to create an mapping in Elastisearch. Because atm, I didn't configured something in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [December 11, 2015, 6:47am UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/8 "2015-12-11T06:47:25Z")

</div>

> The question is also, if I need to specify a type or have to create an mapping in Elastisearch. Because atm, I didn't configured something in Elasticsearch.

You don't have to, but often the default mappings aren't a perfect fit.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:18am UTC](https://discuss.elastic.co/t/problems-with-parsing-xml-log-files-with-xml-filter-of-logstash/36796/9 "2017-07-06T05:18:56Z")

</div>