Exclude log messages in logstash

Elastic Stack Logstash
1

I'm picking up data from log files using filebeat and sending it to Elasticsearch via Logstash. I wanted to exclude few log lines. So can I use an if condition in Logstash. If yes, please share me the format and guide me.

2

I suggest you exclude the lines at Filebeat itself, rather than picking up and sending it to Logstash and then processing there.

exclude_lines edit

A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped. Empty lines are ignored.

If multiline settings are also specified, each multiline message is combined into a single line before the lines are filtered by exclude_lines.

The following example configures Filebeat to drop any lines that start with DBG.

filebeat.inputs: - type: log ... exclude_lines: ['^DBG']

filebeat.inputs:
- type: log
  ...
  exclude_lines: ['^DBG']
1 Like
3

Thanks for the idea.

I will try this solution definitely, but is there a way to do at the Logstash end as well?

4

One way of filtering out unnecessary fields from events in logstash is by using the drop/remove field filter.

This field will remove those fields and only the remaining fields will be mapped or stored in Elasticsearch.

    filter {
      drop {
        remove_field => [ "foo_%{somefield}" ]
      }
    }

You can refer the below doc if you need additional information.

elastic.co

Drop filter plugin | Logstash Reference [8.5] | Elastic

Thanks,
Asish

1 Like
5

Thanks a lot for a quick reply.

I will try it out and will let you know.

1 Like
6

You can use drop for a message, remove_field for one or more fields, and prune filter with white/black list.
To reduce the traffic, remove on the source Filebeat.

7

This will unconditionally delete all events, which is almost certainly not useful. The remove_field will have no effect since a drop filter never calls the filter_matched function from the base filter class, so add_tag, add_field, remove_tag, remove_field are not implemented (they are documented because they can be defined without causing an exception, but they are ignored).

You can use a drop filter with a conditional. For example, if you have parsed a [logLevel] field out of a message you might use

if [logLevel] not in [ "WARN", "ERROR" ] { drop {} }

If you want to drop fields rather than events then use either a prune filter, or a mutate filter with the remove_field option.

8

Thanks, I restricted it at the filebeat end itself as you suggested.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.