Filebeat to ouput which pattern matched from the include_lines list

Elastic Stack Beats
1

We use file beat extensively to capture exception in the log files. We define all the error pattern inside the include_lines in the prospector section in array. This can identify the error pattern and using multi line it aggregates errors lines and sends me the event.

However, the info about which pattern is matched currently from the array of patterns is not shown in the result. It would be nice to show matched_pattern value in the output when include_lines is used.
I also understand when RegEx is used to match we cannot output any match pattern.

2

Hi,

It looks to me that you want to do some extra processing to the message, I think Logstash would be the right place to do it, please check grok documentation to see if it fits your needs: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html

3

Hi, sorry i feel i didn't explain it clearly.

For example in my filebeat config, i have the following...

filebeat.prospectors:

  • paths:
    • /var/log/myapp/*.log
      include_lines: ['^ERROR', '^WARN']

when it outputs, i expect filebeat to include attribute like "mattched_pattern":"ERROR" or "mattched_pattern":"WARN"... since each events will be matched based on this include_lines.....

currently it outputs only message attribute, which gives the complete error detail.. but i cant identify which pattern matched from the list....

4

exekias, i have explained my req below. please take a look.

5

Yes, I think this is not possible with filebeat, as we don't normally do processing on messages, but shipping. You need to use Logstash for this, take a look to the given documentation

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.