Multiple regex matches in a single line

Elastic Stack Logstash
1

Hi All,

I want to do multiple regex searches in a single line via logstash.

My log file
DEBUG - PerfLogging input_count:1 ,className:value,date:16-Aug-2016 07:13:40 ,TimeTaken:0.428 ,output_count:1

I want to get the value CLASS=value
and
DATE=16-Aug-2016 07:13:40
via the gork filter.

By fiddling around grok debugger, I found the following patterns.

finding class by a regex pattern search.
(?<CLASS>(?<=className:)\w{1,})

{
  "CLASS": [
    [
      "value"
    ]
  ]
}

Similarly for date,
(?<DATE>(?<=date:)[^,]{1,})

{
  "DATE": [
    [
      "16-Aug-2016 07:13:40 "
    ]
  ]
}

But I am not able to find both the fields at once.
(?<CLASS>(?<=className:)\w{1,}) (?<DATE>(?<=date:)[^,]{1,})

No Matches

Question 1.
How should I get this working for more than one field, Once the filter starts working for two fields I would like to get all the values in similar fashion ( input_count etc .. ).

Question 2.
How do I ignore all the lines in the log file from processing which don't have PerfLogging keyword ?

Thanks,
Ayush

2

How should I get this working for more than one field, Once the filter starts working for two fields I would like to get all the values in similar fashion ( input_count etc .. ).

I suggest you use the kv filter for this instead of grok.

How do I ignore all the lines in the log file from processing which don't have PerfLogging keyword ?

The conditional filter

if [message] !~ /PerfLogging/ {
  drop { }
}

would do.

3

Thanks Magnus for the quick reply.
I am able to get the data from the logs using the KV filter as desired.

But I see the type of the field, for example input_size stored as "string" in logstash
I want to store input_size as "integer", TimeTaken as "double", date as datetype and so on,

Is there any to typecast these fields ?

Thanks,
Ayush.

4

Use the mutate filter's convert option.

5

works just like I wanted it to, Thanks,