Can we add fields value based on regex patterns in include_lines for the same log

[krishanagrawal (Krishan)]

I am parsing a log file which has error from 3 different process.
I have added the input file in path and have added the 3 regex pattern in include line unique to 3 processes.
Can I add field column which will be populated with the log line based on the regex pattern it matched in include_lines tag?

eg:

logs are:
[p1_err] adc....
[p2_err] adc....
[p3_warn] adc....
[p1_warn] adc....
[p3_err] adc....
....

in my filebeat.yml

filebeat.inputs:

• type: log
  paths:
  • <log_file>
    include_lines: ['p1_err', 'p2_err', 'p3_err']

I want to see the count of p1_err,p2_err,p3_err lines in output on kibana.

[steffens (Steffen Siering)]

include_lines is just for prefiltering, it's no parser.

You can either go schemaless and use filtering on the "message" field in kibana UI in order to count events matching a pattern or use grok (Logstash or Ingest Node), or dissect (Filebeat or Logstash or Ingest Node) to do some minimal parsing. E.g. see Filebeat dissect docs.