vinoossk
(vinoo subramaniam kanakasabapathy)
September 21, 2016, 10:11am
1
"####<Sep 19, 2016 6:54:32 AM EDT> <dbqlgbsap01.us.dnb.com > <dnb_domain01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <0000LT25JRt1baV0U3R5Fs1NrwFW000002> <1474282472208> <User connection factory "SOAJMSModule!XmlSchemaChangeNotificationConnectionFactory" is started.> "
####<Sep 19, 2016 6:54:32 AM EDT> <dbqlgbsap01.us.dnb.com > <dnb_domain01> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <> <> <0000LT25JRt1baV0U3R5Fs1NrwFW000002> <1474282472210> <Creating WorkManager from "wmSOAWorkManager" WorkManagerMBean for application "UMSJMSSystemResource">
<soap:Body xmlns:ns1="http://services.dnb.com/GlobalBatchServiceV1.0 ">
ns5:BatchDetail
ns5:ApplicationID Test</ns5:ApplicationID>
ns5:ApplicationBatchID Test</ns5:ApplicationBatchID>
ns5:MessageTimeStamp 2015-12-13T01:14:42-08:00</ns5:MessageTimeStamp>
</ns5:BatchDetail>
</soap:Body>
i want to separate transaction XML and log message and have to parse ApplicationID by passing input as ApplicationBatchID
magnusbaeck
(Magnus Bäck)
September 21, 2016, 11:43am
2
There's no XML in your post. Please format the log snippet as preformatted text using the </>
toolbar button.
vinoossk
(vinoo subramaniam kanakasabapathy)
September 22, 2016, 6:15am
3
####<Sep 16, 2016 3:00:58 AM EDT> <Info> <ALSB Logging> <dbilsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <4098825ab168b478:-2ebacfe3:1570e6b61b8:-8000-000000000001f813> <1474009258784> <BEA-000000> < [A&L Pipeline1, A&L Pipeline_response, A&L ResponseStage, RESPONSE] Toolkit ES AuditRequest: <soapenv:Body xmlns:adap="http://www.dnb.com/AdapterProductService" xmlns:gad="http://services.dnb.com/GADataMain" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<AuditTransaction xmlns="http://services.dnb.com/AuditServiceV2.0" xmlns:ser="http://services.dnb.com/ServiceHeaderV2.0">
<ser:ServiceHeader>
<ser:ApplicationDetail>
<ser:ApplicationID>3</ser:ApplicationID>
<ser:ApplicationTransactionID>adraaaasddfd1212</ser:ApplicationTransactionID>
<ser:ApplicationRequestTimeStamp>2013-08-14T06:44:14.58</ser:ApplicationRequestTimeStamp>
</ser:ApplicationDetail> </ser:ServiceHeader>
</AuditTransaction >
vinoossk
(vinoo subramaniam kanakasabapathy)
September 22, 2016, 6:18am
4
In this log message, I should get ApplicationID by filtering ApplicationTransactionID. I came to see XML{} function but dont know to use effectively for my requirement.
Have you used a multiline codec to get all lines of the message into a single Logstash event? Once you're there, use a grok filter to extract the timestamp and whatever else comes before the XML into their own fields, including the XML that should go into one field. Then use the xml filter to processs the XML field and extract the pieces you're interested in.
vinoossk
(vinoo subramaniam kanakasabapathy)
September 22, 2016, 7:20am
6
Thanks a lot magnus. Able to parse now.
vinoossk
(vinoo subramaniam kanakasabapathy)
September 27, 2016, 10:54am
7
I am getting some part of logs attached to filtered XML
filter
{
multiline {
pattern => "^####"
negate => true
what => "next"}
grok {
match => ["message", "####<%{DATA:wls_timestamp}> <%{WORD:severity}> <%{DATA:wls_topic}> <%{DATA:hostname}> <%{WORD:server}> %{GREEDYDATA:logmessage}"]
}
if "_grokparsefailure" in [tags]
{
drop {}
}
if "multiline" in [tags] {
xml
{
source => "logmessage"
store_xml => "false"
remove_namespaces => "true"
xpath => [ "/GetDGXPacketRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
xpath => [ "/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
xpath => [ "/GetXTEDataRequest/GetDataRequest/TransactionDetail/ApplicationTransactionID/text()", "APP_ID" ]
xpath => [ "/AuditTransaction/ServiceHeader/TransactionDetail/TransactionID/text()", "APP_ID" ]
add_tag => XML_IN
}}
if "_xmlparsefailure" in [tags]
{
drop {}
}
if "XML_IN" not in [tags]
{
drop {}
}
if "XML_IN" in [tags]{
mutate{
add_field => ["MSG_XML", "%{logmessage}"]
add_field => ["HOST_PROC", "%{host}"]
}}
mutate
{
remove_field => [ "@version", "path", "host", "wls_topic", "logmessage", "server", "severity", "message", "hostname"]
}}
vinoossk
(vinoo subramaniam kanakasabapathy)
September 27, 2016, 10:57am
8
**My Output has still some soap tags**
"MSG_XML" => " <pcm:GetDataRequest ServiceVersionNumber=\"\" xmlns:pcm=\"http://services.dnb.com/PCMDataServiceV2.0\">\n <TransactionDetail>\n <ApplicationTransactionID>Sample_915</ApplicationTransactionID>\n <SubmittingOfficeID>XML dnbToolkit</SubmittingOfficeID>\n </TransactionDetail>\n <GetDataRequestDetail>\n <InquiryDetail>\n <DUNSNumber>001368083</DUNSNumber>\n <CountryISOAlpha2Code>US</CountryISOAlpha2Code>\n </InquiryDetail>\n <ProductSpecification>\n <TradeUpIndicator>true</TradeUpIndicator>\n <ProductDataXPathText>Product/Organization/Assessment/SupplierEvaluationRiskScore</ProductDataXPathText>\n </InquiryReferenceDetail>\n </GetDataRequestDetail>\n </pcm:GetDataRequest>\n</soap-env:Body>>\n####<Sep 15, 2016 8:11:34 AM EDT> <Info> <ALSB Logging> <dbqlsteap01.us.dnb.com> <Mac01_mngd01> <[ACTIVE] ExecuteThread: '135' for queue: 'weblogic.kernel.Default (self-tuning)'> <<anonymous>> <> <25b3a1ebf1f8cf6d:658f2dca:1572c2cbe81:-7ffb-0000000000018cba> <1473941494796> <BEA-000000> < [RouteToPCMDataService, null, null, REQUEST] Header GDP Flow: <soapenv:Header xmlns:soapenv=\"http://schemas.xmlsoap.org/soap/envelope/\">",
Part trying to remove : from ####
vinoossk
(vinoo subramaniam kanakasabapathy)
October 3, 2016, 10:20am
10
Is it possible to fetch single tag constantly like (irrespective of positions) across all XML's instead of XPATH ??