# Exclude log messages in logstash

**URL:** <https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829>\
**Category:** Logstash\
**Created:** [January 10, 2023, 12:13pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829 "2023-01-10T12:13:37Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![tejal\_kubde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tejal_kubde/32/99611_2.png) [@tejal\_kubde](https://discuss.elastic.co/u/tejal_kubde)\
**Post date:** [January 10, 2023, 12:13pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/1 "2023-01-10T12:13:37Z")

</div>

I'm picking up data from log files using filebeat and sending it to Elasticsearch via Logstash. I wanted to exclude few log lines. So can I use an if condition in Logstash. If yes, please share me the format and guide me.

---

<div class="post-metadata">

**Author:** ![dadiasish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadiasish/32/114221_2.png) [@dadiasish](https://discuss.elastic.co/u/dadiasish)\
**Post date:** [January 10, 2023, 12:17pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/2 "2023-01-10T12:17:45Z")

</div>

I suggest you exclude the lines at Filebeat itself, rather than picking up and sending it to Logstash and then processing there.

##### `exclude_lines` [edit](https://github.com/elastic/beats/edit/8.5/filebeat/docs/inputs/input-common-harvester-options.asciidoc)

A list of regular expressions to match the lines that you want Filebeat to exclude. Filebeat drops any lines that match a regular expression in the list. By default, no lines are dropped. Empty lines are ignored.

If [multiline](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html#multiline) settings are also specified, each multiline message is combined into a single line before the lines are filtered by `exclude_lines`.

The following example configures Filebeat to drop any lines that start with `DBG`.

filebeat.inputs: - type: log ... exclude\_lines: ['^DBG']

```auto
filebeat.inputs:
- type: log
  ...
  exclude_lines: ['^DBG']

```

---

<div class="post-metadata">

**Author:** ![tejal\_kubde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tejal_kubde/32/99611_2.png) [@tejal\_kubde](https://discuss.elastic.co/u/tejal_kubde)\
**Post date:** [January 10, 2023, 12:19pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/3 "2023-01-10T12:19:56Z")

</div>

Thanks for the idea.

I will try this solution definitely, but is there a way to do at the Logstash end as well?

---

<div class="post-metadata">

**Author:** ![dadiasish](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadiasish/32/114221_2.png) [@dadiasish](https://discuss.elastic.co/u/dadiasish)\
**Post date:** [January 10, 2023, 12:20pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/4 "2023-01-10T12:20:41Z")

</div>

One way of filtering out unnecessary fields from events in logstash is by using the drop/remove field filter.

This field will remove those fields and only the remaining fields will be mapped or stored in Elasticsearch.

```auto
    filter {
      drop {
        remove_field => ["foo_%{somefield}"]
      }
    }

```

You can refer the below doc if you need additional information.

![](https://us1.discourse-cdn.com/elastic/original/3X/f/a/facc403660137bf4bf60f4ba3f206830fa8604fc.png)[elastic.co](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-remove_field)

![](https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt280217a63b82a734/6202d3378b1f312528798412/elastic-logo.svg)

### [Drop filter plugin | Logstash Reference [8.5] | Elastic ](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html#plugins-filters-drop-remove_field)

Thanks,  
Asish

---

<div class="post-metadata">

**Author:** ![tejal\_kubde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tejal_kubde/32/99611_2.png) [@tejal\_kubde](https://discuss.elastic.co/u/tejal_kubde)\
**Post date:** [January 10, 2023, 12:21pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/5 "2023-01-10T12:21:47Z")

</div>

Thanks a lot for a quick reply.

I will try it out and will let you know.

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [January 10, 2023, 1:41pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/6 "2023-01-10T13:41:59Z")

</div>

You can use **drop** for a message, **remove\_field** for one or more fields, and prune filter with white/black list.  
To reduce the traffic, remove on the source Filebeat.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 10, 2023, 6:45pm UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/7 "2023-01-10T18:45:43Z")

</div>

> [@dadiasish](#):
>
> ```auto
> filter {
> drop {
> remove_field => ["foo_%{somefield}"]
> }
> }
> 
> ```

This will unconditionally delete all events, which is almost certainly not useful. The remove\_field will have no effect since a drop filter never calls the filter\_matched function from the base filter class, so add\_tag, add\_field, remove\_tag, remove\_field are not implemented (they are documented because they can be defined without causing an exception, but they are ignored).

You can use a drop filter with a conditional. For example, if you have parsed a [logLevel] field out of a message you might use

if [logLevel] not in ["WARN", "ERROR"] { drop {} }

If you want to drop fields rather than events then use either a prune filter, or a mutate filter with the remove\_field option.

---

<div class="post-metadata">

**Author:** ![tejal\_kubde](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tejal_kubde/32/99611_2.png) [@tejal\_kubde](https://discuss.elastic.co/u/tejal_kubde)\
**Post date:** [January 11, 2023, 6:31am UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/8 "2023-01-11T06:31:50Z")

</div>

Thanks, I restricted it at the filebeat end itself as you suggested.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 8, 2023, 6:31am UTC](https://discuss.elastic.co/t/exclude-log-messages-in-logstash/322829/9 "2023-02-08T06:31:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.