Separate log message and transaction message

Elastic Stack Logstash
#1

"####<Sep 19, 2016 6:54:32 AM EDT> <dbqlgbsap01.us.dnb.com> <dnb_domain01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <0000LT25JRt1baV0U3R5Fs1NrwFW000002> <1474282472208> <User connection factory "SOAJMSModule!XmlSchemaChangeNotificationConnectionFactory" is started.> "
####<Sep 19, 2016 6:54:32 AM EDT> <dbqlgbsap01.us.dnb.com> <dnb_domain01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <0000LT25JRt1baV0U3R5Fs1NrwFW000002> <1474282472210> <Creating WorkManager from "wmSOAWorkManager" WorkManagerMBean for application "UMSJMSSystemResource">
<soap:Body xmlns:ns1="http://services.dnb.com/GlobalBatchServiceV1.0">

ns5:BatchDetail
ns5:ApplicationIDTest</ns5:ApplicationID>
ns5:ApplicationBatchIDTest</ns5:ApplicationBatchID>
ns5:MessageTimeStamp2015-12-13T01:14:42-08:00</ns5:MessageTimeStamp>
</ns5:BatchDetail>

</soap:Body>

i want to separate transaction XML and log message and have to parse ApplicationID by passing input as ApplicationBatchID

#2

There's no XML in your post. Please format the log snippet as preformatted text using the </> toolbar button.

#3 
####<Sep 16, 2016 3:00:58 AM EDT> <Info> <ALSB Logging> <dbilsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4098825ab168b478:-2ebacfe3:1570e6b61b8:-8000-000000000001f813> <1474009258784> <BEA-000000> < [A&L Pipeline1, A&L Pipeline_response, A&L ResponseStage, RESPONSE] Toolkit ES AuditRequest: <soapenv:Body xmlns:adap="http://www.dnb.com/AdapterProductService" xmlns:gad="http://services.dnb.com/GADataMain" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <AuditTransaction xmlns="http://services.dnb.com/AuditServiceV2.0" xmlns:ser="http://services.dnb.com/ServiceHeaderV2.0">
    <ser:ServiceHeader>
      <ser:ApplicationDetail>
        <ser:ApplicationID>3</ser:ApplicationID>
        <ser:ApplicationTransactionID>adraaaasddfd1212</ser:ApplicationTransactionID>
        <ser:ApplicationRequestTimeStamp>2013-08-14T06:44:14.58</ser:ApplicationRequestTimeStamp>
      </ser:ApplicationDetail>    </ser:ServiceHeader>
</AuditTransaction >
#4

In this log message, I should get ApplicationID by filtering ApplicationTransactionID. I came to see XML{} function but dont know to use effectively for my requirement.

#5

Have you used a multiline codec to get all lines of the message into a single Logstash event? Once you're there, use a grok filter to extract the timestamp and whatever else comes before the XML into their own fields, including the XML that should go into one field. Then use the xml filter to processs the XML field and extract the pieces you're interested in.

#6

Thanks a lot magnus. Able to parse now.

#7 
I am getting some part of logs attached to filtered XML

    filter
    {
    multiline {
     pattern => "^####"
       negate => true
        what => "next"}
    grok {
    match => ["message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{DATA:hostname}> <%{WORD:server}> %{GREEDYDATA:logmessage}"]
    }
    if "_grokparsefailure"  in [tags]
    {
    drop {}
    }
    if "multiline" in [tags] {
    xml
    {
    source => "logmessage"
    store_xml => "false"
    remove_namespaces => "true"
    xpath => [ "/GetDGXPacketRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
    xpath => [ "/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
    xpath => [ "/GetXTEDataRequest/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
    xpath => [ "/AuditTransaction/ServiceHeader/TransactionDetail/TransactionID/text()", "APP_ID" ]
    add_tag => XML_IN
    }}
    if "_xmlparsefailure" in [tags]
    {
    drop {}
    }
    if "XML_IN" not in [tags]
    {
    drop {}
    }
    if  "XML_IN" in [tags]{
    mutate{
    add_field => ["MSG_XML", "%{logmessage}"]
    add_field => ["HOST_PROC", "%{host}"]
    }}
    mutate
    {
    remove_field => [ "@version", "path", "host", "wls_topic", "logmessage", "server", "severity", "message", "hostname"]
    }}
#8 
**My Output has still some soap tags**
"MSG_XML" => "  <pcm:GetDataRequest ServiceVersionNumber=\"\" xmlns:pcm=\"http://services.dnb.com/PCMDataServiceV2.0\">\n    <TransactionDetail>\n      <ApplicationTransactionID>Sample_915</ApplicationTransactionID>\n      <SubmittingOfficeID>XML dnbToolkit</SubmittingOfficeID>\n    </TransactionDetail>\n    <GetDataRequestDetail>\n      <InquiryDetail>\n        <DUNSNumber>001368083</DUNSNumber>\n        <CountryISOAlpha2Code>US</CountryISOAlpha2Code>\n      </InquiryDetail>\n      <ProductSpecification>\n        <TradeUpIndicator>true</TradeUpIndicator>\n        <ProductDataXPathText>Product/Organization/Assessment/SupplierEvaluationRiskScore</ProductDataXPathText>\n   </InquiryReferenceDetail>\n    </GetDataRequestDetail>\n  </pcm:GetDataRequest>\n</soap-env:Body>>\n####<Sep 15, 2016 8:11:34 AM EDT> <Info> <ALSB Logging> <dbqlsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '135' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <25b3a1ebf1f8cf6d:658f2dca:1572c2cbe81:-7ffb-0000000000018cba> <1473941494796> <BEA-000000> < [RouteToPCMDataService, null, null, REQUEST] Header GDP Flow: <soapenv:Header xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">",

Part trying to remove : from ####
#10

Is it possible to fetch single tag constantly like (irrespective of positions) across all XML's instead of XPATH ??

#11