_grokparsefailure help required ASAP

Elastic Stack Logstash
1

what could be the regex for this log
['Estimated', '904699466068', '508587356765', 'LineItem', '61133026369107517924193456', 'AWS CloudTrail', '16091348', '215570231', '1256864', 'APS1-FreeEventsRecorded', 'None', '', 'N', '0.0 per free event recorded in Asia Pacific (Singapore) region', '2017-03-01 00:00:00', '2017-03-01 01:00:00', '67.00000000', '0.0000000000', '0.00000000', '0.0000000000', '0.00000000', '', '', '', '', '', '', '', '', '', '', '', '']

this is an csv file and I am using grok filter. The columns are skewed so I have to use grok for matching

My regex is this
grok {

    match =>{ "message" => "\[%{GREEDYDATA:InvoiceID}\,%{GREEDYDATA:PayerAccountId}\,%{GREEDYDATA:LinkedAccountId}\,%{GREEDYDATA:RecordType},%{GREEDYDATA:RecordId}\,%{GREEDYDATA:ProductName}\,
      %{GREEDYDATA:RateId}\,%{GREEDYDATA:SubscriptionId}\,%{GREEDYDATA:PricingPlanId}\,%{GREEDYDATA:UsageType}\,%{GREEDYDATA:Operation}\,%{GREEDYDATA:AvailabilityZone}\,%{GREEDYDATA:ReservedInstance}\,%{GREEDYDATA:ItemDescription}\,%{GREEDYDATA:UsageStartDate}\,%{GREEDYDATA:UsageEndDate}\,%{GREEDYDATA:UsageQuantity}\,%{GREEDYDATA:BlendedRate}\,%{GREEDYDATA:BlendedCost}\,%{GREEDYDATA:UnBlendedRate}\,%{GREEDYDATA:UnBlendedCost}\,%{GREEDYDATA:ResourceId}\,%{GREEDYDATA:aws:cloudformation:logical-id}\,%{GREEDYDATA:aws:cloudformation:stack-id}\,%{GREEDYDATA:aws:cloudformation:stack-name}\,%{GREEDYDATA:user:Application}\,%{GREEDYDATA:user:Project}\,%{GREEDYDATA:user:Stack}\,%{GREEDYDATA:user:cso_rollup1}\,%{GREEDYDATA:user:cso_rollup2}\,%{GREEDYDATA:user:cso_rollup3}\,%{GREEDYDATA:user:owner}\,%{GREEDYDATA:user:poc}\]" }
  }

It is the amazon billing log

2

Hi Kaustuv,

Instead of using the grok filter, you could use the csv filter and specify the names of the columns that are fixed. Any additional columns will be auto numbered.

Could you provide your config file, sample log file and the screenshot of the error?

1 Like
3

The columns are skewed

What do you mean by this?

I don't have time to debug this, but I'll make two general suggestions

  • Don't use multiple GREEDYDATA patterns. As long as the values won't contain quotes you should be able to say e.g. (?<name-of-field>[^']*) to match zero or more non-quotes.
  • Start with the simplest possible expression, e.g. \[%{GREEDYDATA:InvoiceID}\, and verify that it works. Add more and more until things stop working. Be systematic.
1 Like
4

Thanks gautam and magnusbaeck it was a regex error. I debugged it and got the answer
Regex I used
grok {

    match =>{ "message" => "\[%{GREEDYDATA:InvoiceID}\,%{GREEDYDATA:PayerAccountId}\,%{GREEDYDATA:LinkedAccountId}\,%{GREEDYDATA:RecordType}\,%{GREEDYDATA:RecordId}\,%{GREEDYDATA:ProductName}\,%{GREEDYDATA:RateId}\,%{GREEDYDATA:SubscriptionId}\,%{GREEDYDATA:PricingPlanId}\,%{GREEDYDATA:UsageType}\,%{GREEDYDATA:Operation}\,%{GREEDYDATA:AvailabilityZone}\,%{GREEDYDATA:ReservedInstance}\,%{GREEDYDATA:ItemDescription}\,%{GREEDYDATA:UsageStartDate}\,%{GREEDYDATA:UsageEndDate}\,%{GREEDYDATA:UsageQuantity}\,%{GREEDYDATA:BlendedRate}\,%{GREEDYDATA:BlendedCost}\,%{GREEDYDATA:UnBlendedRate}\,%{GREEDYDATA:UnBlendedCost}\,%{GREEDYDATA:ResourceId}\,%{GREEDYDATA:aws:cloudformation:logical-id}\,%{GREEDYDATA:aws:cloudformation:stack-id}\,%{GREEDYDATA:aws:cloudformation:stack-name}\,%{GREEDYDATA:user:Application}\,%{GREEDYDATA:user:Project}\,%{GREEDYDATA:user:Stack}\,%{GREEDYDATA:user:cso_rollup1}\,%{GREEDYDATA:user:cso_rollup2}\,%{GREEDYDATA:user:cso_rollup3}\,%{GREEDYDATA:user:owner}\,%{GREEDYDATA:user:poc}\]" }
  }
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.