Compare message if it's similar to others

Elastic Stack Logstash
1

Hi,
filebeat.yml configuration:

multiline.pattern: \d\d/\d\d/\d\d\d\d\s\d\d:\d\d:\d\d\s\d\d\d-\sMAIN\sEXCEPTION\b
multiline.negate: true
multiline.match: after
multiline.max_lines: 7

Logstash configuration:

input {
    beats {
	    type => "test"
        port => "5044"
    }
}
output{
    stdout {
    codec => rubydebug
  }
    email {
    to => "aa@gmail.com"
    via => 'smtp'
	address => 'smtp.gmail.com'
    domain => 'smtp.gmail.com'
	from => 'jo@gmail.com'
    authentication => "plain"
    username => "jo@gmail.com"
	password => "pass"
	subject => 'Alert '
	body => "%{message}"
	port => 25
	use_tls => true
    }    
    elasticsearch {
        hosts => ["localhost:9200"]        
    }
}

Imagine that in one file i have the same message like more than 50 for example will send me a 50 emails so i want to add a filter bloc or somthing to just send me one email contains this message if it's similar to others.

Any help would be sincerely appreciate!
Thanks.

2

I suppose you need some Aggregate Filter Plugin.

4

Thanks for the reply,
i don't understand what should i use,
I just want if there is the word "EXCEPTION" in this file send me just one mail contains there is EXCEPTION in this file ...

Any help would be sincerely appreciate!
Thanks.

5

I suppose that is something different from the original post...

6

It's not that different just i want to receive one email if i found the message is the same that the next message

7

You could use a throttle filter to tag the events and then use a conditional in the output.

1 Like
8

Thanks Badger for the reply,
I read about throttle filter now and i tried but still can't find the solution :confused:

input {
    beats {
	    type => "test"
        port => "5044"
    }
}
filter {
   if !("EXCEPTION" in [message]) {drop{}}
   throttle {
        before_count => 1
        after_count => 3
        period => 600
        max_age => 1200
        key => "%{message}"
        add_tag => "throttled"
      }
      if "throttled" in [tags] {
        drop { }
      }      
}
output{
    stdout {
    codec => rubydebug
  }
   if "throttled" not in [tags] {
    email {
    to => "aa@gmail.com"
    via => 'smtp'
	address => 'smtp.gmail.com'
    domain => 'smtp.gmail.com'
	from => 'jo@gmail.com'
    authentication => "plain"
    username => "jo@gmail.com"
	password => "pass"
	subject => '--- Alert ---'
	body => "%{message}"
	port => 25
	use_tls => true
    }
    }
    elasticsearch {
        hosts => ["localhost:9200"]        
    }

It still gives me more than one message
If you can tell me what is the problem please and thanks.

9

That will tag any messages before the first (there aren't any), and any messages after the third, so there will be 3 messages you do not drop. If you just want the first then use

before_count => -1
after_count => 1
2 Likes
10

Thanks Badger it works :heart:,
That was my bad was testing with a different message the time make me wrong...
Last question, i give a file at first and it works good but if i give another file after 10min for example did not work

filter {
   if !("IOException" in [message]) {drop{}}
   throttle {
        before_count => -1
        after_count => 1
        period => 600
        max_age => 1200
        key => "%{host}%{message}"
        add_tag => "throttled"
      }
      if "throttled" in [tags] {
        drop { }
      }  
}

What should i update to make this works when i give a new file anytime?

Thanks!

11

If you want to make the throttle dependent on the name of the file that events came from you would have to use an aggregate filter rather than throttle. You have set period to 600 so all events except one in a ten minute period will be throttled.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.