Unable to process log file using logstash

This is a sample of my log file

wlapp@ieo2wlzsoat09$ soalogs
wlapp@ieo2wlzsoat09$ grep a231326a System.log

<l:event dateTime="2019-04-10 07:58:13.669" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="BEGIN" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http">customerBill</http:relative-URI><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="debugFlag" value="true" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload/></ResourceLog></l:event>

<l:event dateTime="2019-04-10 07:58:13.702" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INFO" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></l:event>

<l:event dateTime="2019-04-10 07:58:13.748" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="OUTBOUND" eventStatus="" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http"/><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="SOATransactionID" value="a231326a-c71b-479c-b3e2-535c40dc0a1a" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></payload></ResourceLog></l:event>

<l:event dateTime="2019-04-10 07:58:19.821" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INBOUND" eventStatus="SUCCESS" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>

<l:event dateTime="2019-04-10 07:58:19.824" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="END" eventStatus="SUCCESS" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>

wlapp@ieo2wlzsoat09$

this is my configuration file

input {

file {
path => "D:\customerBill_logs.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "<l:event dateTime=""
negate => true
what => "previous"
auto_flush_interval => 1
charset => "ISO-8859-1"
}
}
}

filter {
grok {
match => {"message" => "<l:event dateTime="%{DATA:dateTime}" layerName="%{DATA:layerName}" processName="%{DATA:processName}" eventType="%{DATA:eventType}" eventStatus="%{DATA:eventStatus}" eventCode="%{DATA:eventCode}" outboundServiceName="%{DATA:outboundServiceName}" serverHostIP="%{DATA:serverHostIP}" applicationID="%{DATA:applicationID}" providerID="%{DATA:providerID}" originatorIP="%{DATA:originatorIP}" SOAConsumerTransactionID="%{DATA:SOAConsumerTransactionID}" SOATransactionID="%{DATA:ID}">%{GREEDYDATA:response}</l:event>"}
}
if "_grokparsefailure" in [tags] {
drop { }
}

}

output {

elasticsearch {
hosts => ["localhost:9200"]
}
stdout
{
codec => rubydebug
}
}

Logstash isn't writing data into elastic search. I'm not getting any errors either.Is there something wrong with this conf

Use forward slash rather than backslash in the path option of the file input.

No Change .

Remove the drop {} filter and check if you are getting _grokparsefailure tags.

No i am not getting it

What does a sample event look like in rubydebug?

Even if i remove this part from output i'm not able to get the output in ES
stdout
{
codec => rubydebug
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.