# Logstash to filter and drop all strings mentioned in exception.txt file

**URL:** <https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107>\
**Category:** Logstash\
**Created:** [January 30, 2020, 5:41am UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107 "2020-01-30T05:41:05Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anirban\_Banerjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anirban_banerjee/32/61699_2.png) [@Anirban\_Banerjee](https://discuss.elastic.co/u/Anirban_Banerjee)\
**Post date:** [January 30, 2020, 5:41am UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/1 "2020-01-30T05:41:05Z")

</div>

Hello

We are trying to see the feasibility of using logstash and icinga (check\_logfiles plugins) in tandem to monitor postgres logs.

The idea is we have a file called ignore\_alert.txt. It has 100s of strings which the admin is not interested in receiving alerts.

> $ cat ignore\_alert.txt  
> (01000)  
> (0100C)  
> (01008)  
> (01003)  
> (01007)  
> (01006)  
> (01004)  
> (01P01)  
> (02000)  
> (02001)  
> (03000)  
> (08000)  
> (08003)  
> (08006)  
> (08001)  
> (08004)  
> (08007)  
> (08P01)  
> (09000)  
> (0A000)  
> (0B000)  
> (0F000)  
> (0F001)  
> (0L000)  
> (0LP01)  
> (0P000)  
> (0Z000)  
> (0Z002)

Why we chose the ignore (exclusion) process than inclusion string is because every time postgres comes with new version - some new strings are introduced which may be crucial. So we want to monitor everything unless it has been confirmed by Admin as not required

So we want to read from postgres logfile filter out anything which matches string in ignore\_alert.txt

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [January 30, 2020, 7:30am UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/2 "2020-01-30T07:30:51Z")

</div>

Hi

If you could get your "ignore\_alert" info in one single line and into an environmental variable in your logstash system, then you could just use it in a filter:

```auto
[...]
mutate {
   add_field {
      IGNORE => ${<your_environment_variable_here>:<default_value __just_in_case__ not_required>}
   }

if [ALERTCODE] in [IGNORE] {
  drop {}
}
[...]

```

Hope this helps

---

<div class="post-metadata">

**Author:** ![Anirban\_Banerjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anirban_banerjee/32/61699_2.png) [@Anirban\_Banerjee](https://discuss.elastic.co/u/Anirban_Banerjee)\
**Post date:** [January 30, 2020, 7:46am UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/3 "2020-01-30T07:46:35Z")

</div>

> [@ITIC](#):
>
> your\_environment\_variable\_here

Thank You for your response

Well the ignore\_alert.txt file is few hundred lines. It has alert IDs as shown above or sometimes strings as well like "wrong statement" with space in between etc etc. this is just too big to bring in 1 single line. Also it is expected that Admins will add new patterns at the end of ignore\_alert.txt file - if he doesn't want to be alerted on some newly discovered pattern. Just wondering if this is even feasible solution considering I cannot just hog on to lot of cpu memory just to process this for every line in logs

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [January 30, 2020, 8:26am UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/4 "2020-01-30T08:26:20Z")

</div>

Hi

You are right, this would be feasible only for a small(ish) list. A more general solution could maybe be implemented using a [`ruby{}`](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html) filter.

Inside the ruby filter you could check your exceptions file and fill a boolean variable, then use an `if` to `drop{}` the rejects.

```auto
if [REJECT] > 0 {
  drop {}
}

```

Hope this helps.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 30, 2020, 4:33pm UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/5 "2020-01-30T16:33:56Z")

</div>

You could do it in a [ruby script](https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html). In the register function load the file into an instance variable array. Then in the filter function do

```
m = event.get("message")
a.each { |x|
    if m.include? x
        event.cancel
    end
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2020, 4:34pm UTC](https://discuss.elastic.co/t/logstash-to-filter-and-drop-all-strings-mentioned-in-exception-txt-file/217107/6 "2020-02-27T16:34:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.