Unable to process log file using logstash

Elastic Stack Logstash
1

This is a sample of my log file

wlapp@ieo2wlzsoat09$ soalogs
wlapp@ieo2wlzsoat09$ grep a231326a System.log

<l:event dateTime="2019-04-10 07:58:13.669" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="BEGIN" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http">customerBill</http:relative-URI><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="debugFlag" value="true" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload/></ResourceLog></l:event>

<l:event dateTime="2019-04-10 07:58:13.702" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INFO" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></l:event>

<l:event dateTime="2019-04-10 07:58:13.748" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="OUTBOUND" eventStatus="" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http"/><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="SOATransactionID" value="a231326a-c71b-479c-b3e2-535c40dc0a1a" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></payload></ResourceLog></l:event>

<l:event dateTime="2019-04-10 07:58:19.821" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INBOUND" eventStatus="SUCCESS" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>

<l:event dateTime="2019-04-10 07:58:19.824" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="END" eventStatus="SUCCESS" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>

wlapp@ieo2wlzsoat09$

this is my configuration file

input {

file {
path => "D:\customerBill_logs.log"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline
{
pattern => "<l:event dateTime=""
negate => true
what => "previous"
auto_flush_interval => 1
charset => "ISO-8859-1"
}
}
}

filter {
grok {
match => {"message" => "<l:event dateTime="%{DATA:dateTime}" layerName="%{DATA:layerName}" processName="%{DATA:processName}" eventType="%{DATA:eventType}" eventStatus="%{DATA:eventStatus}" eventCode="%{DATA:eventCode}" outboundServiceName="%{DATA:outboundServiceName}" serverHostIP="%{DATA:serverHostIP}" applicationID="%{DATA:applicationID}" providerID="%{DATA:providerID}" originatorIP="%{DATA:originatorIP}" SOAConsumerTransactionID="%{DATA:SOAConsumerTransactionID}" SOATransactionID="%{DATA:ID}">%{GREEDYDATA:response}</l:event>"}
}
if "_grokparsefailure" in [tags] {
drop { }
}

}

output {

elasticsearch {
hosts => ["localhost:9200"]
}
stdout
{
codec => rubydebug
}
}

Logstash isn't writing data into Elasticsearch. I'm not getting any errors either.Is there something wrong with this conf

2

Use forward slash rather than backslash in the path option of the file input.

3

No Change .

4

Remove the drop {} filter and check if you are getting _grokparsefailure tags.

5

No i am not getting it

6

What does a sample event look like in rubydebug?

7

Even if i remove this part from output i'm not able to get the output in ES
stdout
{
codec => rubydebug
}

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.