GROK Parse failure for log file pattern

Elastic Stack Logstash
1
  1. 2019-05-12 15:57:15.446 [localhost] [Services] [Request Id : RT1242124202120519027049] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [1242,124202,RT1242124202120519027049] Exit Add_SaleLineItem: Output [" Transaction_Status=In process Transaction_Total=�0.00 Item_Count=1 RefundExpiryDate=Wed Jun 26 23:59:59 BST 2019 ageRestrictedItemDeclinedByCashier=false ageRestrictionByPassByCashier= promotionId={rrrLog.transaction.customerOrderTransaction.PromotionSavingsSummary.savingsList.0-promotionId}"]
  2. 2019-05-12 15:57:15.788 [localhost] [Services] [Request Id : 98626a7b-cbce-40d3-9fff-31455aba7f53] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [,,] Enter ActiveTransaction: Input [""]
  3. 2019-05-12 15:57:15.805 [localhost] [Services] [Request Id : 98626a7b-cbce-40d3-9fff-31455aba7f53] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [,,] Exit ActiveTransaction: Output ["TransactionId =RT1242124202120519027049 SequenceNumber =270 TransactionCurrencyCode =GBP TransactionBeginDateTime =Sun May 12 15:57:14 BST 2019" ]
  4. 2019-05-12 15:57:18.884 [localhost] [Services] [Request Id : dc89f23d-7c43-4cc3-b47b-0f9c6392b690] [User Id : ] ERROR Log:104 - Servicesstore.heartbeat.serviceimpl.DbServerHealthCheckServiceImpl : in DbServerHealthCheckServiceImpl class: checkHealth method errorjdbc/PrimaryDataSource -- service jboss.naming.context.java.jboss.jdbc.PrimaryDataSource
  5. 2019-05-12 15:59:49.415 [localhost] [Services] [Request Id : 1531e5bd-6e67-439e-9b9a-a44f7687cab8] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [1242,124202,] Exit LookupLastTransaction: Output ["Transaction_id =RT1242124202120519027049 TransactionDateTime =rrrlog.DateTimeCommonData@55eb23bb TotalQuantity =4 TenderTotal = GrandTotal = TotalSalesAmount =�10.99 TotalReturnAmount =�15.00"]
  6. 2019-05-12 15:59:49.415 [localhost] [Services] [Request Id : 1531e5bd-6e67-439e-9b9a-a44f7687cab8] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [1242,124202,] Exit LookupLastTransaction: Output ["Transaction_id =RT1242124202120519027049 TransactionDateTime =rrrlog.DateTimeCommonData@55eb23bb TotalQuantity =10 TenderTotal = GrandTotal = TotalSalesAmount =�152.99 TotalReturnAmount =�86.00"]
  7. 2019-05-12 16:01:15.767 [localhost] [Services] [Request Id : 91406c10-6abd-493b-99ba-f2acad4c947f] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [,,] Enter ActiveTransaction: Input [""]
  8. 2019-05-13 07:27:34.311 [localhost] [Services] [Request Id : c3489430-78ff-4db4-bf05-0539187d582d] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [1242,124202,RT1242124202130519000589] Enter Add_Cash_Tender: Input ["Transaction_Type =RetailTransaction Transaction_Status= Operator_ID = TEST@sales.com Tender_line_no = 0 Tender_type = Cash tenderAmount = 36.52 Currency_Code = GBP"]
  9. 2019-05-12 16:01:15.785 [localhost] [Services] [Request Id : 91406c10-6abd-493b-99ba-f2acad4c947f] [User Id : TEST@sales.com] INFO Services.aspect.AspectController:120 - [,,] Exit ActiveTransaction: Output ["TransactionId =RT1242124202120519027293 SequenceNumber =272 TransactionCurrencyCode =GBP TransactionBeginDateTime =Sun May 12 16:01:15 BST 2019" ]

Hi All

Above is my log message printed from server.I need to slice the data from line numbers 5,6 & 8.

This is my GROK filter
filter {
grok {
match => [
"message" , "%{GREEDYDATA}TotalQuantity%{SPACE}=%{DATA}%{NUMBER:TotalQuantity:int}%{GREEDYDATA}TotalSalesAmount%{SPACE}=%{DATA}%{NUMBER:TotalSalesAmount:float}%{SPACE}TotalReturnAmount%{SPACE}=%{DATA}%{NUMBER:TotalReturnAmount:float}",
"message" , "%{GREEDYDATA}Cash tenderAmount%{SPACE}=%{DATA}%{NUMBER:CashTenderAmount:float}" ]
remove_field => ["message"]
}
}

and this works fine only if log message exactly like line numbers 5,6 & 8 are printed or else I am getting "tags" => [
[0] "_grokparsefailure"
],
message in logstash .Since they keywords strings will be printed in between the logs I am not sure how to use this filters to parse the entire logs and extract the value which I want.And my log file will not be in exact pattern as like GROK.

Any help is much appreciated

2

you should parse the data from beginning, you are ignoring beginning data as GREEDYDATA and you are losing the information.
I observed your logs, that contains the information how they are different from each others
for example: at line no. 5 and 6 it contains "Exit LookupLastTransaction" and line 8 has "Enter Add_Cash_Tender" may be these are the types, so first parse the data till this point and keep remaining data in a single field.

afterwards you make condition based on the types and then parse the remaining filed(by using kv filter or grok filter).

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.