Correct configuration file for XML input


(Abhisek Mohanty) #1

My log look like this which is "|" separated.

The log consist of id, status, request, response fields.

The request and response are present in XML data which are "|" separated.

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

And I have use configuration file like this :

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    sincedb_path => "/dev/null"

    multiline {
  pattern => "</soapenv:Envelope>"
  what => "previous"
  negate => "true"
}
  }
}
filter {
  grok {
    match => [ "message", "%{DATA:method_id}|%{WORD:method_type}|%{GREEDYDATA:data}" ]
  } 
}

output {
   elasticsearch {
     hosts => "http://localhost:XXXX"
     index => "req_res"
  }
stdout {}
}

But it is not working. Please let me know what changes should be made in configuration file.

My output should look like this:

ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response - <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

Help me with the configuration file..


(Magnus Bäck) #2

I can spot two bugs:

  • If you want to match a literal "|" in a regular expression you need to escape it.
  • The multiline configuration is most likely wrong. You'll want to adjust the expression so it matches the first line of the message, i.e. something like ^%{INT}\|%{WORD}\|<soapenv.

While not directly related to your problem, the grok expression is inefficient. See my multiline suggestion above for an example of how to start the expression.


(Abhisek Mohanty) #3

Thanks magnus for your reply,

My issue was resolved with your multiline expression

But there is a new issue that i'm stuck with, which is how will i parse for multiple xml log.

As you can see, my question has only one XML log data and which is working perfectly fine with your given pattern.

But when i add multiple log then it parse it as a 1 log data.

for ex :
this is how my 1st log starts

and my 2nd log is like this :

220000|write|<soapenv:Envelope xmlns:....


(Magnus Bäck) #4

I'm not 100% sure what you mean so please show a complete example of a log you want to process.


(Abhisek Mohanty) #5

Thanks for reply

This is my first log:

110000|read|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

and the 2nd log :

213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

similarly there are "n" no. of logs

so using the above config file i'm able to map the 1st log but not the 2nd one.


(Magnus Bäck) #6

And in what way is it failing for the second set of log lines? What does your configuration look like?


(Abhisek Mohanty) #7

my configuration look like this :slightly_smiling_face:

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
     start_position => "beginning"
        codec => multiline
  {
   pattern => "^%{INT}\|%{WORD}\|<soapenv:Envelope*>\|</soapenv:Envelope"
   negate => true
   what => "next"
  }
  }
}
filter {
  grok {
    match => [ "message", "%{INT:method_id}\|%{WORD:method_type}\|%{GREEDYDATA:request}\|%{GREEDYDATA:response}"]
}
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

Currently my logs are being mapped like this:

ID- 110000

status- read

request - 
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>| <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>
213000|write|<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:web="http://webservices.lookup.sdp.bharti.ibm.com"><soapenv:Header/><soapenv:Bod<web:getLookUpServiceDetails>
<getLookUpService>
<serviceRequester>XXX</serviceRequester>
<lineOfBusiness>XXX</lineOfBusiness>
<lookupAttribute>
<searchAttrValue>XXX</searchAttrValue>
</lookupAttribute>
</getLookUpService>
</web:getLookUpServiceDetails>
</soapenv:Body>
</soapenv:Envelope>

response -<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Body>
<ns:getLookUpServiceDetailsResponse xmlns:ns="http://webservices.lookup.sdp.bharti.ibm.com">
<getLookUpServiceReturn>
<errorInfo>
<ErrorCode/>
<ErrorMessage/>
</errorInfo>
<lookupResponseList>
<mapEntry>
<attributeName>region</attributeName>
<attributeValue>["XXX"]</attributeValue>
</mapEntry>
</lookupResponseList>
</getLookUpServiceReturn>
</ns:getLookUpServiceDetailsResponse>
</soapenv:Body>
</soapenv:Envelope>

(Magnus Bäck) #8

Use what => "previous" in your multiline codec.


(Abhisek Mohanty) #9

Still getting the same issue.


(Abhisek Mohanty) #10

I tried a new config file

input {
  file {
    path => "/opt/test5/practice_new/data.xml"
    start_position => "beginning"
    codec => multiline {
            pattern => "^%{NUMBER:method_id}\|%{DATA:method_type}\|<soapenv:Envelope>"
            negate => true
            what => "previous"
        }
  }
}
filter {
  grok {
    match => [ "(?m)^(?<method_id>\d+)\|(?<method_type>\w+)\|(?<request>[^|]*)\|(?<response>[^|\n]*(?:\n(?!\d+\|)[^|\n]*)*)" ]
  }
}

output {
   elasticsearch {
     hosts => "http://localhost:9200"
     index => "req_res"
  }
stdout {}
}

Here i have used regular expression to match my logs.

but for this config it matches for only 1st log and does't not match form 2nd log onwards.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.