Correct configuration file for XML input

abhisek · August 17, 2017, 12:55pm

My log look like this which is "|" separated.

The log consist of id, status, request, response fields.

The request and response are present in XML data which are "|" separated.

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

And I have use configuration file like this :

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    sincedb_path => "/dev/null"

    multiline {
  pattern => "</soapenv:Envelope>"
  what => "previous"
  negate => "true"
}
  }
}
filter {
  grok {
    match => [ "message", "%{DATA:method_id}|%{WORD:method_type}|%{GREEDYDATA:data}" ]
  } 
}

output {
   elasticsearch {
     hosts => "http://localhost:XXXX"
     index => "req_res"
  }
stdout {}
}

But it is not working. Please let me know what changes should be made in configuration file.

My output should look like this:

ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response - <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

Help me with the configuration file..

magnusbaeck · August 20, 2017, 8:39pm

I can spot two bugs:

If you want to match a literal "|" in a regular expression you need to escape it.
The multiline configuration is most likely wrong. You'll want to adjust the expression so it matches the first line of the message, i.e. something like ^%{INT}\|%{WORD}\|<soapenv.

While not directly related to your problem, the grok expression is inefficient. See my multiline suggestion above for an example of how to start the expression.

abhisek · August 22, 2017, 6:57am

Thanks magnus for your reply,

My issue was resolved with your multiline expression

But there is a new issue that i'm stuck with, which is how will i parse for multiple xml log.

As you can see, my question has only one XML log data and which is working perfectly fine with your given pattern.

But when i add multiple log then it parse it as a 1 log data.

for ex :
this is how my 1st log starts

and my 2nd log is like this :

220000|write|<soapenv:Envelope xmlns:....

magnusbaeck · August 24, 2017, 6:06am

I'm not 100% sure what you mean so please show a complete example of a log you want to process.

abhisek · August 24, 2017, 6:31am

Thanks for reply

This is my first log:

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

and the 2nd log :

213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

similarly there are "n" no. of logs

so using the above config file i'm able to map the 1st log but not the 2nd one.

magnusbaeck · August 24, 2017, 2:36pm

And in what way is it failing for the second set of log lines? What does your configuration look like?

abhisek · August 28, 2017, 12:57pm

my configuration look like this

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
     start_position => "beginning"
        codec => multiline
  {
   pattern => "^%{INT}\|%{WORD}\|<soapenv:Envelope*>\|</soapenv:Envelope"
   negate => true
   what => "next"
  }
  }
}
filter {
  grok {
    match => [ "message", "%{INT:method_id}\|%{WORD:method_type}\|%{GREEDYDATA:request}\|%{GREEDYDATA:response}"]
}
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

Currently my logs are being mapped like this:

ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>| <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response -<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

magnusbaeck · August 28, 2017, 1:59pm

Use what => "previous" in your multiline codec.

abhisek · August 30, 2017, 12:06pm

Still getting the same issue.

abhisek · August 31, 2017, 4:52am

I tried a new config file

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    start_position => "beginning"
    codec => multiline {
            pattern => "^%{NUMBER:method_id}\|%{DATA:method_type}\|<soapenv:Envelope>"
            negate => true
            what => "previous"
        }
  }
}
filter {
  grok {
    match => [ "(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request>[^|]*)\|(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)" ]
  }
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

Here i have used regular expression to match my logs.

but for this config it matches for only 1st log and does't not match form 2nd log onwards.

system · September 28, 2017, 4:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.