# Unable to process log file using logstash

**URL:** <https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992>\
**Category:** Logstash\
**Created:** [April 23, 2019, 9:03am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992 "2019-04-23T09:03:52Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\
**Post date:** [April 23, 2019, 9:03am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/1 "2019-04-23T09:03:52Z")

</div>

This is a sample of my log file

> ```
> wlapp@ieo2wlzsoat09$ soalogs
> wlapp@ieo2wlzsoat09$ grep a231326a System.log
> 
> <l:event dateTime="2019-04-10 07:58:13.669" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="BEGIN" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http">customerBill</http:relative-URI><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="debugFlag" value="true" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload/></ResourceLog></l:event>
> 
> <l:event dateTime="2019-04-10 07:58:13.702" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INFO" eventStatus="" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></l:event>
> 
> <l:event dateTime="2019-04-10 07:58:13.748" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="OUTBOUND" eventStatus="" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><ResourceLog><http:relative-URI xmlns:http="http://www.bea.com/wli/sb/transports/http"/><http:http-method xmlns:http="http://www.bea.com/wli/sb/transports/http">GET</http:http-method><http:query-parameters xmlns:http="http://www.bea.com/wli/sb/transports/http"><http:parameter name="billingAccount.id" value="700001367"/></http:query-parameters><customHttpHeaders><tran:user-header name="applicationID" value="appID" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/><tran:user-header name="SOATransactionID" value="a231326a-c71b-479c-b3e2-535c40dc0a1a" xmlns:http="http://www.bea.com/wli/sb/transports/http" xmlns:tran="http://www.bea.com/wli/sb/transports"/></customHttpHeaders><payload><cus:customerBillRequest xmlns:cus="http://www.three.middleware.services.com/3im/schema/customerbillmanagementdata_v1_0"><cus:billingAccount><cus:id>700001367</cus:id></cus:billingAccount></cus:customerBillRequest></payload></ResourceLog></l:event>
> 
> <l:event dateTime="2019-04-10 07:58:19.821" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="INBOUND" eventStatus="SUCCESS" eventCode="" outboundServiceName="CustomerBillManagementAdapter_V1_0.customerBill" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>
> 
> <l:event dateTime="2019-04-10 07:58:19.824" layerName="OSB" processName="CustomerBillManagement_V1_0.customerBill" eventType="END" eventStatus="SUCCESS" eventCode="" outboundServiceName="" serverHostIP="ieo2wlzsoat09/10.142.4.23/soa_server1" applicationID="appID" providerID="threeie" originatorIP="10.142.15.15" SOAConsumerTransactionID="" SOATransactionID="a231326a-c71b-479c-b3e2-535c40dc0a1a"><soapenv:Body xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">{"SOATransactionID":"a231326a-c71b-479c-b3e2-535c40dc0a1a","customerBill":{"id":"700001367","runType":"onCycle","category":"normal","billDate":null,"paymentDueDate":null,"appliedPayment":[{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":879.0}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":1299.99}}},{"payment":{"paymentDate":"2018-11-16","amount":{"unit":"EUR","value":729.0}}}]}}</soapenv:Body></l:event>
> 
> wlapp@ieo2wlzsoat09$
> 
> ```

this is my configuration file

> input {
> 
> file {  
> path =\> "D:\customerBill\_logs.log"  
> start\_position =\> "beginning"  
> sincedb\_path =\> "/dev/null"  
> codec =\> multiline  
> {  
> pattern =\> "\<l:event dateTime=""  
> negate =\> true  
> what =\> "previous"  
> auto\_flush\_interval =\> 1  
> charset =\> "ISO-8859-1"  
> }   
> }   
> }
> 
> filter {  
> grok {  
> match =\> {"message" =\> "\<l:event dateTime="%{DATA:dateTime}" layerName="%{DATA:layerName}" processName="%{DATA:processName}" eventType="%{DATA:eventType}" eventStatus="%{DATA:eventStatus}" eventCode="%{DATA:eventCode}" outboundServiceName="%{DATA:outboundServiceName}" serverHostIP="%{DATA:serverHostIP}" applicationID="%{DATA:applicationID}" providerID="%{DATA:providerID}" originatorIP="%{DATA:originatorIP}" SOAConsumerTransactionID="%{DATA:SOAConsumerTransactionID}" SOATransactionID="%{DATA:ID}"\>%{GREEDYDATA:response}\</l:event\>"}  
> }  
> if "\_grokparsefailure" in [tags] {  
> drop { }  
> }
> 
> }
> 
> output {
> 
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> }  
> stdout  
> {  
> codec =\> rubydebug  
> }  
> }

Logstash isn't writing data into Elasticsearch. I'm not getting any errors either.Is there something wrong with this conf

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2019, 11:24am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/2 "2019-04-23T11:24:35Z")

</div>

> [@sunilkumar](#):
>
> path =\> "D:\customerBill\_logs.log"

Use forward slash rather than backslash in the path option of the file input.

---

<div class="post-metadata">

**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\
**Post date:** [April 23, 2019, 11:47am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/3 "2019-04-23T11:47:27Z")

</div>

No Change .

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2019, 11:50am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/4 "2019-04-23T11:50:20Z")

</div>

Remove the drop {} filter and check if you are getting \_grokparsefailure tags.

---

<div class="post-metadata">

**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\
**Post date:** [April 23, 2019, 11:54am UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/5 "2019-04-23T11:54:28Z")

</div>

No i am not getting it

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 23, 2019, 12:08pm UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/6 "2019-04-23T12:08:42Z")

</div>

What does a sample event look like in rubydebug?

---

<div class="post-metadata">

**Author:** ![sunilkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilkumar/32/28675_2.png) [@sunilkumar](https://discuss.elastic.co/u/sunilkumar)\
**Post date:** [April 23, 2019, 12:21pm UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/7 "2019-04-23T12:21:16Z")

</div>

Even if i remove this part from output i'm not able to get the output in ES  
stdout  
{  
codec =\> rubydebug  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2019, 12:21pm UTC](https://discuss.elastic.co/t/unable-to-process-log-file-using-logstash/177992/8 "2019-05-21T12:21:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.