# How to parse irregular xml logs

**URL:** <https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646>\
**Category:** Logstash\
**Created:** [August 16, 2018, 7:49am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646 "2018-08-16T07:49:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 16, 2018, 7:49am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/1 "2018-08-16T07:49:31Z")

</div>

Hi

How to parse irregular xml logs

sample log is shown below.

DEBUG 2018/08/16 13:01:17.307842 Response XML:- \<Header version="1.0"msg\_type="SERVICE\_REQUEST"\>GR\_IFA\_IN\_B2C\_127\_0\_0\_1\_01\<Operator\_details operator\_w\_name="svt" name="SVN"\>\</Operator\_details\>\<Control\_AccountInfo\>\</Control\_AccountInfo\>\<PG\_AccountInfo\>\</PG\_AccountInfo\>\<Service\_Request name="get\_limit"\>\</Service\_Request\>\<Duration\_Info Initial\_time="1534404677" Subsystem\_name="IFA\_IN"\>\</Duration\_Info\>\<Duration\_Info Final\_time="1534404677307" Initial\_time="1534404677154" Total\_time="153" Subsystem\_name="SM"\>\</Duration\_Info\>\<Duration\_Info Final\_time="1534404677301" Initial\_time="1534404677159" Total\_time="142" Subsystem\_name="TP"\>\</Duration\_Info\>\<Duration\_Info Final\_time="1534404677200" Initial\_time="1534404677164" Total\_time="36" Subsystem\_name="USR"\>\</Duration\_Info\>\<Duration\_Info Final\_time="1534404677293" Initial\_time="1534404677211" Total\_time="82" Subsystem\_name="WL"\>\</Duration\_Info\>

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2018, 8:53am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/2 "2018-08-16T08:53:42Z")

</div>

What do you mean by irregular, i.e. what exactly are you having problems with?

---

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 16, 2018, 9:45am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/4 "2018-08-16T09:45:27Z")

</div>

Logs with different values or fields (problem iam facing is to define Xpath in logstash xml filter because there is no fixed values or fields).

TRACE 2018/08/16 12:44:33.496034 Response XML is \<zero\_System\_Interface version="1.0"\>\<SystemService\_Response version="1.0" name="GetTxnInfo"\>\<Response\_Info\>0030No record found.01DECLINED000000000696Transaction Record not found\</Response\_Info\>\</SystemService\_Response\>\</zero\_System\_Interface\>\</receiver  
TRACE 2018/08/16 12:45:33.496084 Response XML is \<zero\_version="1.0"\>\<zero\_System\_Interface version="1.0"\>\<SystemService\_Response version="1.0" name="GetTxnInfo"\>0030No record found.01DECLINED000000000696Transaction Record not found\</SystemService\_Response\>\</zero\_System\_Interface\>\</zero\_version\>

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2018, 10:25am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/5 "2018-08-16T10:25:37Z")

</div>

Well, if the XML has a varying schema you'll have problems defining XPath expressions. Perhaps you should just parse the whole XML into the event?

---

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 16, 2018, 10:59am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/6 "2018-08-16T10:59:55Z")

</div>

While parsing whole XML into event, fields are missing

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2018, 11:02am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/7 "2018-08-16T11:02:53Z")

</div>

What XML are you parsing? What result do you get?

---

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 16, 2018, 11:18am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/8 "2018-08-16T11:18:02Z")

</div>

Iam using this filter....

filter {  
xml {  
source =\> "message"  
target =\> "parser"  
}  
}

My logs are

@timestamp August 20th 2018, 09:51:42.715  
t @version 1  
t \_id JpmRVWUBQofrep64Wv4us  
t \_index filebeat-2018.08.20

# \_score -

t \_type doc  
t beat.hostname grcentos  
t beat.name grcentos  
t beat.version 6.3.2  
t host.name grcentos  
t input.type log  
t message TRACE 2018/08/16 12:44:33.496034 Response XML is \<receiver\_INTERFACE version="1.0"\>\<receiver\_System\_Interface version="1.0"\>\<SystemService\_Response version="1.0" name="GetTxnInfo"\>\<Response\_Info\>0030No record found.01DECLINED000000000696Transaction Record not found\</Response\_Info\>\</SystemService\_Response\>\</receiver\_System\_Interface\>\</receiver\_INTERFACE\>

# offset 1,575

? parser.receiver\_System\_Interface {  
"SystemService\_Response": [  
{  
"version": "1.0",  
"name": "GetTxnInfo",  
"Response\_Info": [  
{  
"HostDescription": [  
"No record found."  
],  
"HostCode": [  
"0030"  
],  
"ClassCode": [  
"01"  
],  
"SystemDescription": [  
"Transaction Record not found"  
],  
"SystemCode": [  
"000000000696"  
],  
"ClassDescription": [  
"DECLINED"  
]  
}  
]  
}  
],  
"version": "1.0"  
}  
? parser.version 1.0  
t prospector.type log  
t source /var/log/test.log  
t tags beats\_input\_codec\_plain\_applied

---

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 20, 2018, 4:38am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/9 "2018-08-20T04:38:35Z")

</div>

Hello Magnus please reply on above query

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 7:05am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/10 "2018-08-20T07:05:16Z")

</div>

Okay, so it appears to parse the XML without any problems. What's the desired outcome?

---

<div class="post-metadata">

**Author:** ![dinesh1](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@dinesh1](https://discuss.elastic.co/u/dinesh1)\
**Post date:** [August 20, 2018, 7:21am UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/11 "2018-08-20T07:21:33Z")

</div>

There is no fields without xpath...

I need fields like as shown below from above parsed xml logs witout Xpath

ClassCode: 01  
SystemDescription:Transaction Record not found  
SystemCode:000000000696

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 20, 2018, 12:45pm UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/12 "2018-08-20T12:45:03Z")

</div>

The fields are there, they're just not rendered in the same way in Kibana. You can move fields with a mutate filter, but since the XML schema isn't fixed but somehow dynamic you'll have to use a ruby filter. In what way are the XML documents different?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 17, 2018, 12:45pm UTC](https://discuss.elastic.co/t/how-to-parse-irregular-xml-logs/144646/13 "2018-09-17T12:45:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.