# Compare message if it's similar to others
**URL:** https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826
**Category:** Logstash
**Created:** [February 10, 2022, 9:35am UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826 "2022-02-10T09:35:31Z")
**Posts on this page:** 11
**Page:** 1
### Author:  [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)
#### Post date: [February 10, 2022, 9:35am UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/1 "2022-02-10T09:35:31Z")
Hi,
filebeat.yml configuration:
```auto
multiline.pattern: \d\d/\d\d/\d\d\d\d\s\d\d:\d\d:\d\d\s\d\d\d-\sMAIN\sEXCEPTION\b
multiline.negate: true
multiline.match: after
multiline.max_lines: 7
```
Logstash configuration:
```auto
input {
beats {
type => "test"
port => "5044"
}
}
output{
stdout {
codec => rubydebug
}
email {
to => "aa@gmail.com"
via => 'smtp'
address => 'smtp.gmail.com'
domain => 'smtp.gmail.com'
from => 'jo@gmail.com'
authentication => "plain"
username => "jo@gmail.com"
password => "pass"
subject => 'Alert '
body => "%{message}"
port => 25
use_tls => true
}
elasticsearch {
hosts => ["localhost:9200"]
}
}
```
Imagine that in one file i have the same message like more than 50 for example will send me a 50 emails so i want to add a filter bloc or somthing to just send me one email contains this message if it's similar to others.
Any help would be sincerely appreciate!
Thanks.
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [February 10, 2022, 12:34pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/2 "2022-02-10T12:34:51Z")
I suppose you need some [Aggregate Filter Plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html).
---
### Author:  [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)
#### Post date: [February 10, 2022, 1:15pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/4 "2022-02-10T13:15:49Z")
Thanks for the reply,
i don't understand what should i use,
I just want if there is the word "EXCEPTION" in this file send me just one mail contains there is EXCEPTION in this file ...
Any help would be sincerely appreciate!
Thanks.
---
### Author:  [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)
#### Post date: [February 10, 2022, 2:47pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/5 "2022-02-10T14:47:09Z")
I suppose that is something different from the original post...
---
### Author:  [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)
#### Post date: [February 10, 2022, 6:38pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/6 "2022-02-10T18:38:15Z")
It's not that different just i want to receive one email if i found the message is the same that the next message
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 10, 2022, 7:12pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/7 "2022-02-10T19:12:06Z")
You could use a [throttle](https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html) filter to tag the events and then use a conditional in the output.
---
### Author:  [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)
#### Post date: [February 10, 2022, 8:47pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/8 "2022-02-10T20:47:25Z")
Thanks Badger for the reply,
I read about throttle filter now and i tried but still can't find the solution 😕
```auto
input {
beats {
type => "test"
port => "5044"
}
}
filter {
if !("EXCEPTION" in [message]) {drop{}}
throttle {
before_count => 1
after_count => 3
period => 600
max_age => 1200
key => "%{message}"
add_tag => "throttled"
}
if "throttled" in [tags] {
drop { }
}
}
output{
stdout {
codec => rubydebug
}
if "throttled" not in [tags] {
email {
to => "aa@gmail.com"
via => 'smtp'
address => 'smtp.gmail.com'
domain => 'smtp.gmail.com'
from => 'jo@gmail.com'
authentication => "plain"
username => "jo@gmail.com"
password => "pass"
subject => '--- Alert ---'
body => "%{message}"
port => 25
use_tls => true
}
}
elasticsearch {
hosts => ["localhost:9200"]
}
```
It still gives me more than one message
If you can tell me what is the problem please and thanks.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 10, 2022, 9:01pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/9 "2022-02-10T21:01:01Z")
> [@alex\_vermex](#):
>
> ```auto
> before_count => 1
> after_count => 3
>
> ```
That will tag any messages before the first (there aren't any), and any messages after the third, so there will be 3 messages you do not drop. If you just want the first then use
```
before_count => -1
after_count => 1
```
---
### Author:  [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)
#### Post date: [February 11, 2022, 8:49am UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/10 "2022-02-11T08:49:19Z")
Thanks Badger it works ❤,
That was my bad was testing with a different message the time make me wrong...
Last question, i give a file at first and it works good but if i give another file after 10min for example did not work
```auto
filter {
if !("IOException" in [message]) {drop{}}
throttle {
before_count => -1
after_count => 1
period => 600
max_age => 1200
key => "%{host}%{message}"
add_tag => "throttled"
}
if "throttled" in [tags] {
drop { }
}
}
```
What should i update to make this works when i give a new file anytime?
Thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 11, 2022, 4:08pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/11 "2022-02-11T16:08:42Z")
> [@alex\_vermex](#):
>
> What should i update to make this works when i give a new file anytime?
If you want to make the throttle dependent on the name of the file that events came from you would have to use an aggregate filter rather than throttle. You have set period to 600 so all events except one in a ten minute period will be throttled.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 11, 2022, 4:08pm UTC](https://discuss.elastic.co/t/compare-message-if-its-similar-to-others/296826/12 "2022-03-11T16:08:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.