# Separate log message and transaction message

**URL:** <https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101>\
**Category:** Logstash\
**Created:** [September 21, 2016, 10:11am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101 "2016-09-21T10:11:08Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 21, 2016, 10:11am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/1 "2016-09-21T10:11:09Z")

</div>

"####\<Sep 19, 2016 6:54:32 AM EDT\> \<[dbqlgbsap01.us.dnb.com](http://dbqlgbsap01.us.dnb.com)\> \<dnb\_domain01\> \<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'\> \<\> \<\> \<0000LT25JRt1baV0U3R5Fs1NrwFW000002\> \<1474282472208\> \<User connection factory "SOAJMSModule!XmlSchemaChangeNotificationConnectionFactory" is started.\> "  
####\<Sep 19, 2016 6:54:32 AM EDT\> \<[dbqlgbsap01.us.dnb.com](http://dbqlgbsap01.us.dnb.com)\> \<dnb\_domain01\> \<[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'\> \<\> \<\> \<0000LT25JRt1baV0U3R5Fs1NrwFW000002\> \<1474282472210\> \<Creating WorkManager from "wmSOAWorkManager" WorkManagerMBean for application "UMSJMSSystemResource"\>  
\<soap:Body xmlns:ns1="[http://services.dnb.com/GlobalBatchServiceV1.0](http://services.dnb.com/GlobalBatchServiceV1.0)"\>  
  
ns5:BatchDetail  
ns5:ApplicationIDTest\</ns5:ApplicationID\>  
ns5:ApplicationBatchIDTest\</ns5:ApplicationBatchID\>  
ns5:MessageTimeStamp2015-12-13T01:14:42-08:00\</ns5:MessageTimeStamp\>  
\</ns5:BatchDetail\>  
  
\</soap:Body\>

i want to separate transaction XML and log message and have to parse ApplicationID by passing input as ApplicationBatchID

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 21, 2016, 11:43am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/2 "2016-09-21T11:43:07Z")

</div>

There's no XML in your post. Please format the log snippet as preformatted text using the `</>` toolbar button.

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 22, 2016, 6:15am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/3 "2016-09-22T06:15:36Z")

</div>

```
####<Sep 16, 2016 3:00:58 AM EDT> <Info> <ALSB Logging> <dbilsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4098825ab168b478:-2ebacfe3:1570e6b61b8:-8000-000000000001f813> <1474009258784> <BEA-000000> < [A&L Pipeline1, A&L Pipeline_response, A&L ResponseStage, RESPONSE] Toolkit ES AuditRequest: <soapenv:Body xmlns:adap="http://www.dnb.com/AdapterProductService" xmlns:gad="http://services.dnb.com/GADataMain" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
 <AuditTransaction xmlns="http://services.dnb.com/AuditServiceV2.0" xmlns:ser="http://services.dnb.com/ServiceHeaderV2.0">
    <ser:ServiceHeader>
      <ser:ApplicationDetail>
        <ser:ApplicationID>3</ser:ApplicationID>
        <ser:ApplicationTransactionID>adraaaasddfd1212</ser:ApplicationTransactionID>
        <ser:ApplicationRequestTimeStamp>2013-08-14T06:44:14.58</ser:ApplicationRequestTimeStamp>
      </ser:ApplicationDetail> </ser:ServiceHeader>
</AuditTransaction >
```

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 22, 2016, 6:18am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/4 "2016-09-22T06:18:21Z")

</div>

`In this log message, I should get ApplicationID by filtering ApplicationTransactionID. I came to see XML{} function but dont know to use effectively for my requirement.`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 22, 2016, 6:26am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/5 "2016-09-22T06:26:48Z")

</div>

Have you used a multiline codec to get all lines of the message into a single Logstash event? Once you're there, use a grok filter to extract the timestamp and whatever else comes before the XML into their own fields, including the XML that should go into one field. Then use the xml filter to processs the XML field and extract the pieces you're interested in.

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 22, 2016, 7:20am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/6 "2016-09-22T07:20:04Z")

</div>

Thanks a lot magnus. Able to parse now.

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 27, 2016, 10:54am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/7 "2016-09-27T10:54:40Z")

</div>

```
I am getting some part of logs attached to filtered XML

    filter
    {
    multiline {
     pattern => "^####"
       negate => true
        what => "next"}
    grok {
    match => ["message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{DATA:hostname}> <%{WORD:server}> %{GREEDYDATA:logmessage}"]
    }
    if "_grokparsefailure" in [tags]
    {
    drop {}
    }
    if "multiline" in [tags] {
    xml
    {
    source => "logmessage"
    store_xml => "false"
    remove_namespaces => "true"
    xpath => ["/GetDGXPacketRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID"]
    xpath => ["/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID"]
    xpath => ["/GetXTEDataRequest/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID"]
    xpath => ["/AuditTransaction/ServiceHeader/TransactionDetail/TransactionID/text()", "APP_ID"]
    add_tag => XML_IN
    }}
    if "_xmlparsefailure" in [tags]
    {
    drop {}
    }
    if "XML_IN" not in [tags]
    {
    drop {}
    }
    if "XML_IN" in [tags]{
    mutate{
    add_field => ["MSG_XML", "%{logmessage}"]
    add_field => ["HOST_PROC", "%{host}"]
    }}
    mutate
    {
    remove_field => ["@version", "path", "host", "wls_topic", "logmessage", "server", "severity", "message", "hostname"]
    }}
```

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [September 27, 2016, 10:57am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/8 "2016-09-27T10:57:48Z")

</div>

```
**My Output has still some soap tags**"MSG_XML" => " <pcm:GetDataRequest ServiceVersionNumber=\"\" xmlns:pcm=\"http://services.dnb.com/PCMDataServiceV2.0\">\n <TransactionDetail>\n <ApplicationTransactionID>Sample_915</ApplicationTransactionID>\n <SubmittingOfficeID>XML dnbToolkit</SubmittingOfficeID>\n </TransactionDetail>\n <GetDataRequestDetail>\n <InquiryDetail>\n <DUNSNumber>001368083</DUNSNumber>\n <CountryISOAlpha2Code>US</CountryISOAlpha2Code>\n </InquiryDetail>\n <ProductSpecification>\n <TradeUpIndicator>true</TradeUpIndicator>\n <ProductDataXPathText>Product/Organization/Assessment/SupplierEvaluationRiskScore</ProductDataXPathText>\n </InquiryReferenceDetail>\n </GetDataRequestDetail>\n </pcm:GetDataRequest>\n</soap-env:Body>>\n####<Sep 15, 2016 8:11:34 AM EDT> <Info> <ALSB Logging> <dbqlsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '135' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <25b3a1ebf1f8cf6d:658f2dca:1572c2cbe81:-7ffb-0000000000018cba> <1473941494796> <BEA-000000> < [RouteToPCMDataService, null, null, REQUEST] Header GDP Flow: <soapenv:Header xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">",

Part trying to remove : from ####
```

---

<div class="post-metadata">

**Author:** ![vinoossk](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@vinoossk](https://discuss.elastic.co/u/vinoossk)\
**Post date:** [October 3, 2016, 10:20am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/10 "2016-10-03T10:20:21Z")

</div>

Is it possible to fetch single tag constantly like (irrespective of positions) across all XML's instead of XPATH ??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:36am UTC](https://discuss.elastic.co/t/separate-log-message-and-transaction-message/61101/11 "2017-07-06T04:36:08Z")

</div>

Closed on July 6, 2017, 4:36am UTC