Logstash filter for a log record from filebeats

Elastic Stack Logstash
1

Hi,

I am using ELK stack+filebeats in our project and the application log is integration with filebeats. Filebeat is reading the application log. The log is successfully read by logstash. My question is how to parse the XML inside the log file. The log is inserted in following format below. Please help me to parse the XML from the log and read the elements from the XML using logstash.

Blockquote

2021-08-23 18:48:22.711 INFO [bwEngThread:In-Memory Process Worker-1] com.tibco.bw.palette.generalactivities.Log.Framework.sharedmodule.Log - MSGCODE: <?xml version="1.0" encoding="UTF-8"?>
<tns:Logger_Request xmlns:tns="http://www.ericsson.com/tibco/schema/Logger" xmlns:tib="http://www.tibco.com/bw/xslt/custom-functions">
tns:conversationIdCONVbw0a101j9t</tns:conversationId>
tns:correlationIdCORIDbw0a101j9t</tns:correlationId>
tns:logTimestamp2021-08-23T18:48:22.703+05:30</tns:logTimestamp>
tns:typeSTART</tns:type>
tns:businessReferenceIdABC</tns:businessReferenceId>
tns:systemConsumerABC</tns:systemConsumer>
tns:serviceNameABC</tns:serviceName>
tns:operationNameABC</tns:operationName>
tns:payload<?xml version="1.0" encoding="UTF-8"?><updateCustomerReq xmlns="http://xmlns.ericsson.com/apioperations/UpdateCustomer" xmlns:ns1="http://xmlns.ericsson.com/CDM/Base" xmlns:ns2="http://xmlns.ericsson.com/CDM/Customer" xmlns:ns4="http://xmlns.ericsson.com/CDM/Commons" xmlns:ns3="http://xmlns.ericsson.com/CDM/BSS/Party"><ns1:customerId>MTX12345672</ns1:customerId><collectionInfo><ns2:collectionAgentEmail>collectionAgentEmail@string.com</ns2:collectionAgentEmail><ns2:collectionLeaderEmail>collectionLeaderEmail@string.com</ns2:collectionLeaderEmail></collectionInfo><CustomerAttributes><ns2:maidenNameOfMother>maidenNameOfMother</ns2:maidenNameOfMother><ns2:businessLine>businessLine</ns2:businessLine><ns2:lineOfBusiness>Banking</ns2:lineOfBusiness><ns2:subBusinessLine>Agency and Bureau</ns2:subBusinessLine><ns2:hideDetailPriceFlag>true</ns2:hideDetailPriceFlag><ns2:showContractNumber>true</ns2:showContractNumber><ns2:contractNumberInformation>contractNumberInformation</ns2:contractNumberInformation><ns2:showContractTitle>true</ns2:showContractTitle><ns2:contractTitleInformation>contractTitleInformation</ns2:contractTitleInformation><ns2:showEmailFlag>true</ns2:showEmailFlag><ns2:showPO1Flag>true</ns2:showPO1Flag><ns2:po1Information>po1Information</ns2:po1Information><ns2:showPO2Flag>true</ns2:showPO2Flag><ns2:po2Information>po2Information</ns2:po2Information><ns2:holdBillFlag>true</ns2:holdBillFlag><ns2:contactType>Indosat Reference</ns2:contactType><ns2:invoicingCompany>invoicingCompany</ns2:invoicingCompany><ns2:accountClass>Special Account</ns2:accountClass></CustomerAttributes><customerDemographics><ns2:IndividualName><ns1:accountName>Feryanto</ns1:accountName><ns1:formattedName>Doddy Feryanto</ns1:formattedName><ns1:contactName>Doddy</ns1:contactName><ns1:deliveryName>deliveryName</ns1:deliveryName></ns2:IndividualName><ns2:dateofBirth>2021-06-10</ns2:dateofBirth><ns2:gender>M</ns2:gender><ns2:maritalStatus>MAR</ns2:maritalStatus><ns2:jobDescription>jobDescription</ns2:jobDescription><ns2:nationality>IDN</ns2:nationality><ns2:hobby>hobby</ns2:hobby><ns2:religion>Hindu</ns2:religion><ns2:education>education</ns2:education><ns2:employerName>employerName</ns2:employerName></customerDemographics><identification><DriversLicence><ns3:IDValue>098766543</ns3:IDValue></DriversLicence><socialSecurityNr><ns3:IDValue>112233445</ns3:IDValue></socialSecurityNr><taxNumber><ns3:IDValue>123456</ns3:IDValue></taxNumber><genericIdentification><ns3:IDTypecode>7</ns3:IDTypecode><ns3:IDValue>IDValue</ns3:IDValue><idExpiryDate>29/05/2021</idExpiryDate></genericIdentification></identification><address><ns4:seqNumber>0</ns4:seqNumber><ns4:addressRoleCode>S</ns4:addressRoleCode><ns4:addressLine1>Street No 2</ns4:addressLine1><ns4:addressLine2>Menara Sel.,22,Jl.H.R</ns4:addressLine2><ns4:addressLine3>address line 3</ns4:addressLine3><ns4:contactAddress>contactAddress</ns4:contactAddress><ns4:city>Jakarta</ns4:city><ns4:state>Makassar</ns4:state><ns4:zip>12940</ns4:zip></address><address><ns4:seqNumber>5</ns4:seqNumber><ns4:addressRoleCode>B</ns4:addressRoleCode><ns4:addressLine1>Street No 2</ns4:addressLine1><ns4:addressLine2>Menara Sel.,22,Jl.H.R</ns4:addressLine2><ns4:addressLine3>address line 3</ns4:addressLine3><ns4:contactAddress>contactAddress</ns4:contactAddress><ns4:city>Jakarta</ns4:city><ns4:state>Makassar</ns4:state><ns4:zip>12940</ns4:zip></address><contactMedium><FaxContact><ns4:areaCode>+62</ns4:areaCode><ns4:number>9118256710</ns4:number></FaxContact><homeContact><ns4:number>9118256711</ns4:number></homeContact><workContact><ns4:number>9118256711</ns4:number></workContact><mainContact><ns4:number>9118256712</ns4:number></mainContact><SMSContact><ns4:number>+6289077562411</ns4:number></SMSContact><emailAddress><ns4:eMailAddress>eMailAddress@string.com</ns4:eMailAddress><ns4:eMailAddress1>eMailAddress1@string.com</ns4:eMailAddress1><ns4:eMailAddress2>eMailAddress2@string.com</ns4:eMailAddress2></emailAddress></contactMedium></updateCustomerReq></tns:payload>
tns:Log-LevelAUDIT</tns:Log-Level>
tns:appSpaceABC</tns:appSpace>
tns:appNodeABC</tns:appNode>
tns:engineABC</tns:engine>
tns:appModuleABC</tns:appModule>
</tns:Logger_Request>. JobId [bw0a101j9u], ProcessInstanceId [bw0a101j9u], Activity [Log], Process [framework.sharedmodule.LogProcess], Module [Framework.sharedmodule:1.0.0.20210823170120], Application [LogPrj:1.0].
2021-08-23 18:48:22.803 INFO [Thread-43] com.tibco.thor.frwk.Application - TIBCO-THOR-FRWK-300006: Started BW Application [LogPrj:1.0]

Blockquote

2

Use an xml filter. If you use store_xml => true then you can just pass the filter the whole message and it will dig the XML out of it.

xml {
    source => "message"
    target => "theXML"
    store_xml => true
}

If you want to use xpath then you will need to use dissect (or grok, or mutate+gsub) to extract the XML from the rest of the message. You will also need to specify the namespaces

namespaces => {
    "ns1" => "http://xmlns.ericsson.com/CDM/Base" xmlns:ns2=> "http://xmlns.ericsson.com/CDM/Customer" xmlns:ns4="http://xmlns.ericsson.com/CDM/Commons" xmlns:ns3="http://xmlns.ericsson.com/CDM/BSS/Party"     
}
3

Hi,

Actually the requirement is to extract the XML from the log message. So if you see the log message there is a string before and after the message. How to remove that . Post extraction we can use the XML filter i understand. Can you help with the filter to extract the XML from the log message.
Thanks,
Rabin

4

As I said, if you use store_xml => true there is no need to extract the XML. If store_xml => false then you can use dissect, grok, or mutate+gsub.

5

Hi,

I have tried the process. It is creating separate document rows in elastic db for each element parsed in XML(attached kibana dashboard snapshot

Screenshot 2021-08-26 143530
Screenshot 2021-08-26 1435301494×765 93.4 KB

). This is not the requirement.
We need to parse the XML and for each element create a separate attribute in the single json in same document. for example tns:conversationId, tns:correlationId, tns:logTimestamp etc should be part of same elastic document as separate attributes and same will be visible in kibana accordingly (attached the screenshot 2
Screenshot 2
Screenshot 21477×171 15.3 KB

). I hope iam able to make understand the requirement.

Thanks,
Rabin

6

You will need to use the multiline processing of filebeat to combine those log file lines. If you have questions about that then ask in the beats forum.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.