# Logstash filter for a log record from filebeats

**URL:** <https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228>\
**Category:** Logstash\
**Created:** [August 23, 2021, 1:15pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228 "2021-08-23T13:15:25Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rabin\_Bhattacharya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabin_bhattacharya/32/84115_2.png) [@Rabin\_Bhattacharya](https://discuss.elastic.co/u/Rabin_Bhattacharya)\
**Post date:** [August 23, 2021, 1:15pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/1 "2021-08-23T13:15:25Z")

</div>

Hi,

I am using ELK stack+filebeats in our project and the application log is integration with filebeats. Filebeat is reading the application log. The log is successfully read by logstash. My question is how to parse the XML inside the log file. The log is inserted in following format below. Please help me to parse the XML from the log and read the elements from the XML using logstash.

> Blockquote

2021-08-23 18:48:22.711 INFO [bwEngThread:In-Memory Process Worker-1] com.tibco.bw.palette.generalactivities.Log.Framework.sharedmodule.Log - MSGCODE: \<?xml version="1.0" encoding="UTF-8"?\>  
\<tns:Logger\_Request xmlns:tns="[http://www.ericsson.com/tibco/schema/Logger](http://www.ericsson.com/tibco/schema/Logger)" xmlns:tib="[http://www.tibco.com/bw/xslt/custom-functions](http://www.tibco.com/bw/xslt/custom-functions)"\>  
tns:conversationIdCONVbw0a101j9t\</tns:conversationId\>  
tns:correlationIdCORIDbw0a101j9t\</tns:correlationId\>  
tns:logTimestamp2021-08-23T18:48:22.703+05:30\</tns:logTimestamp\>  
tns:typeSTART\</tns:type\>  
tns:businessReferenceIdABC\</tns:businessReferenceId\>  
tns:systemConsumerABC\</tns:systemConsumer\>  
tns:serviceNameABC\</tns:serviceName\>  
tns:operationNameABC\</tns:operationName\>  
tns:payload\<?xml version="1.0" encoding="UTF-8"?\>\<updateCustomerReq xmlns="[http://xmlns.ericsson.com/apioperations/UpdateCustomer](http://xmlns.ericsson.com/apioperations/UpdateCustomer)" xmlns:ns1="[http://xmlns.ericsson.com/CDM/Base](http://xmlns.ericsson.com/CDM/Base)" xmlns:ns2="[http://xmlns.ericsson.com/CDM/Customer](http://xmlns.ericsson.com/CDM/Customer)" xmlns:ns4="[http://xmlns.ericsson.com/CDM/Commons](http://xmlns.ericsson.com/CDM/Commons)" xmlns:ns3="[http://xmlns.ericsson.com/CDM/BSS/Party](http://xmlns.ericsson.com/CDM/BSS/Party)"\>\<ns1:customerId\>MTX12345672\</ns1:customerId\>\<collectionInfo\>\<ns2:collectionAgentEmail\>collectionAgentEmail@string.com\</ns2:collectionAgentEmail\>\<ns2:collectionLeaderEmail\>collectionLeaderEmail@string.com\</ns2:collectionLeaderEmail\>\</collectionInfo\>\<CustomerAttributes\>\<ns2:maidenNameOfMother\>maidenNameOfMother\</ns2:maidenNameOfMother\>\<ns2:businessLine\>businessLine\</ns2:businessLine\>\<ns2:lineOfBusiness\>Banking\</ns2:lineOfBusiness\>\<ns2:subBusinessLine\>Agency and Bureau\</ns2:subBusinessLine\>\<ns2:hideDetailPriceFlag\>true\</ns2:hideDetailPriceFlag\>\<ns2:showContractNumber\>true\</ns2:showContractNumber\>\<ns2:contractNumberInformation\>contractNumberInformation\</ns2:contractNumberInformation\>\<ns2:showContractTitle\>true\</ns2:showContractTitle\>\<ns2:contractTitleInformation\>contractTitleInformation\</ns2:contractTitleInformation\>\<ns2:showEmailFlag\>true\</ns2:showEmailFlag\>\<ns2:showPO1Flag\>true\</ns2:showPO1Flag\>\<ns2:po1Information\>po1Information\</ns2:po1Information\>\<ns2:showPO2Flag\>true\</ns2:showPO2Flag\>\<ns2:po2Information\>po2Information\</ns2:po2Information\>\<ns2:holdBillFlag\>true\</ns2:holdBillFlag\>\<ns2:contactType\>Indosat Reference\</ns2:contactType\>\<ns2:invoicingCompany\>invoicingCompany\</ns2:invoicingCompany\>\<ns2:accountClass\>Special Account\</ns2:accountClass\>\</CustomerAttributes\>\<customerDemographics\>\<ns2:IndividualName\>\<ns1:accountName\>Feryanto\</ns1:accountName\>\<ns1:formattedName\>Doddy Feryanto\</ns1:formattedName\>\<ns1:contactName\>Doddy\</ns1:contactName\>\<ns1:deliveryName\>deliveryName\</ns1:deliveryName\>\</ns2:IndividualName\>\<ns2:dateofBirth\>2021-06-10\</ns2:dateofBirth\>\<ns2:gender\>M\</ns2:gender\>\<ns2:maritalStatus\>MAR\</ns2:maritalStatus\>\<ns2:jobDescription\>jobDescription\</ns2:jobDescription\>\<ns2:nationality\>IDN\</ns2:nationality\>\<ns2:hobby\>hobby\</ns2:hobby\>\<ns2:religion\>Hindu\</ns2:religion\>\<ns2:education\>education\</ns2:education\>\<ns2:employerName\>employerName\</ns2:employerName\>\</customerDemographics\>\<identification\>\<DriversLicence\>\<ns3:IDValue\>098766543\</ns3:IDValue\>\</DriversLicence\>\<socialSecurityNr\>\<ns3:IDValue\>112233445\</ns3:IDValue\>\</socialSecurityNr\>\<taxNumber\>\<ns3:IDValue\>123456\</ns3:IDValue\>\</taxNumber\>\<genericIdentification\>\<ns3:IDTypecode\>7\</ns3:IDTypecode\>\<ns3:IDValue\>IDValue\</ns3:IDValue\>\<idExpiryDate\>29/05/2021\</idExpiryDate\>\</genericIdentification\>\</identification\>\<address\>\<ns4:seqNumber\>0\</ns4:seqNumber\>\<ns4:addressRoleCode\>S\</ns4:addressRoleCode\>\<ns4:addressLine1\>Street No 2\</ns4:addressLine1\>\<ns4:addressLine2\>Menara Sel.,22,Jl.H.R\</ns4:addressLine2\>\<ns4:addressLine3\>address line 3\</ns4:addressLine3\>\<ns4:contactAddress\>contactAddress\</ns4:contactAddress\>\<ns4:city\>Jakarta\</ns4:city\>\<ns4:state\>Makassar\</ns4:state\>\<ns4:zip\>12940\</ns4:zip\>\</address\>\<address\>\<ns4:seqNumber\>5\</ns4:seqNumber\>\<ns4:addressRoleCode\>B\</ns4:addressRoleCode\>\<ns4:addressLine1\>Street No 2\</ns4:addressLine1\>\<ns4:addressLine2\>Menara Sel.,22,Jl.H.R\</ns4:addressLine2\>\<ns4:addressLine3\>address line 3\</ns4:addressLine3\>\<ns4:contactAddress\>contactAddress\</ns4:contactAddress\>\<ns4:city\>Jakarta\</ns4:city\>\<ns4:state\>Makassar\</ns4:state\>\<ns4:zip\>12940\</ns4:zip\>\</address\>\<contactMedium\>\<FaxContact\>\<ns4:areaCode\>+62\</ns4:areaCode\>\<ns4:number\>9118256710\</ns4:number\>\</FaxContact\>\<homeContact\>\<ns4:number\>9118256711\</ns4:number\>\</homeContact\>\<workContact\>\<ns4:number\>9118256711\</ns4:number\>\</workContact\>\<mainContact\>\<ns4:number\>9118256712\</ns4:number\>\</mainContact\>\<SMSContact\>\<ns4:number\>+6289077562411\</ns4:number\>\</SMSContact\>\<emailAddress\>\<ns4:eMailAddress\>eMailAddress@string.com\</ns4:eMailAddress\>\<ns4:eMailAddress1\>eMailAddress1@string.com\</ns4:eMailAddress1\>\<ns4:eMailAddress2\>eMailAddress2@string.com\</ns4:eMailAddress2\>\</emailAddress\>\</contactMedium\>\</updateCustomerReq\>\</tns:payload\>  
tns:Log-LevelAUDIT\</tns:Log-Level\>  
tns:appSpaceABC\</tns:appSpace\>  
tns:appNodeABC\</tns:appNode\>  
tns:engineABC\</tns:engine\>  
tns:appModuleABC\</tns:appModule\>  
\</tns:Logger\_Request\>. JobId [bw0a101j9u], ProcessInstanceId [bw0a101j9u], Activity [Log], Process [framework.sharedmodule.LogProcess], Module [Framework.sharedmodule:1.0.0.20210823170120], Application [LogPrj:1.0].  
2021-08-23 18:48:22.803 INFO [Thread-43] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [LogPrj:1.0]

> Blockquote

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 23, 2021, 11:23pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/2 "2021-08-23T23:23:02Z")

</div>

Use an xml filter. If you use store\_xml =\> true then you can just pass the filter the whole message and it will dig the XML out of it.

```
xml {
    source => "message"
    target => "theXML"
    store_xml => true
}

```

If you want to use xpath then you will need to use dissect (or grok, or mutate+gsub) to extract the XML from the rest of the message. You will also need to specify the namespaces

```
namespaces => {
    "ns1" => "http://xmlns.ericsson.com/CDM/Base" xmlns:ns2=> "http://xmlns.ericsson.com/CDM/Customer" xmlns:ns4="http://xmlns.ericsson.com/CDM/Commons" xmlns:ns3="http://xmlns.ericsson.com/CDM/BSS/Party"     
}
```

---

<div class="post-metadata">

**Author:** ![Rabin\_Bhattacharya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabin_bhattacharya/32/84115_2.png) [@Rabin\_Bhattacharya](https://discuss.elastic.co/u/Rabin_Bhattacharya)\
**Post date:** [August 24, 2021, 5:30am UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/3 "2021-08-24T05:30:45Z")

</div>

Hi,

Actually the requirement is to extract the XML from the log message. So if you see the log message there is a string before and after the message. How to remove that . Post extraction we can use the XML filter i understand. Can you help with the filter to extract the XML from the log message.  
Thanks,  
Rabin

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 24, 2021, 2:04pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/4 "2021-08-24T14:04:39Z")

</div>

> [@Rabin\_Bhattacharya](#):
>
> Actually the requirement is to extract the XML from the log message. So if you see the log message there is a string before and after the message. How to remove that .

As I said, if you use store\_xml =\> true there is no need to extract the XML. If store\_xml =\> false then you can use [dissect](https://discuss.elastic.co/t/wich-is-the-best-way-to-parse-logs-with-xml-seccion/179556/2), grok, or [mutate+gsub](https://discuss.elastic.co/t/issue-while-parsing-xml-data/196988/4).

---

<div class="post-metadata">

**Author:** ![Rabin\_Bhattacharya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabin_bhattacharya/32/84115_2.png) [@Rabin\_Bhattacharya](https://discuss.elastic.co/u/Rabin_Bhattacharya)\
**Post date:** [August 26, 2021, 9:00am UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/5 "2021-08-26T09:00:06Z")

</div>

> [@Badger](#):
>
> ```auto
> xml {
> source => "message"
> target => "theXML"
> store_xml => true
> }
> 
> ```

Hi,

I have tried the process. It is creating separate document rows in elastic db for each element parsed in XML(attached kibana dashboard snapshot

 ![Screenshot 2021-08-26 143530](https://us1.discourse-cdn.com/elastic/original/3X/5/9/598179ea1ef6a0336d6e8fb1344390904f5a4bf8.png)  
). This is not the requirement.  
We need to parse the XML and for each element create a separate attribute in the single json in same document. for example tns:conversationId, tns:correlationId, tns:logTimestamp etc should be part of same elastic document as separate attributes and same will be visible in kibana accordingly (attached the screenshot 2 ![Screenshot 2](https://us1.discourse-cdn.com/elastic/original/3X/5/7/574e71bd68e7278e834ec488902090ee320e8c1b.png)  
). I hope iam able to make understand the requirement.

Thanks,  
Rabin

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 26, 2021, 3:35pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/6 "2021-08-26T15:35:57Z")

</div>

You will need to use the [multiline processing](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html) of filebeat to combine those log file lines. If you have questions about that then ask in the beats forum.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 23, 2021, 3:36pm UTC](https://discuss.elastic.co/t/logstash-filter-for-a-log-record-from-filebeats/282228/7 "2021-09-23T15:36:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.