Need help on Elastalert

Elastic Stack Elasticsearch
1

I have setup ELK environment. Installed filebeat on few servers and installed logstash on separate single server -10-192-4-253. I have setup Elastalert configuration in logstash server. Below is "/example_frequency.yaml" at "/opt/logstash/python/elastalert" location

type: frequency
index: logstash-*
num_events: 50
timeframe:
hours: 4
filter:

  • term:
    type: "stdout"
    alert:
  • "email"
    alert_text: |
    "ElastAlert has detected suspicious activity for {0} < b>ElastAlert has detected suspicious activity for {0}< /b>"
    At {1}, an {2} error occured. Do something about it!

alert_text_args:

  • email
  • host
  • type
    email:

In the alert mail I am also getting the json code as follow

"ElastAlert has detected suspicious activity for < b>ElastAlert has detected suspicious activity for < /b>"
At ip-10-169-1-48.ec2.internal, an stdout error occured. Do something about it!

At least 50 events occurred between 2016-05-05 03:36 EDT and 2016-05-05 07:36 EDT

(following is json code)
@timestamp: 2016-05-05T11:36:32.022Z
@version: 1
_id: AVSAtIJ5Ydfq7dPgG_SL
_index: logstash-2016.05.05
_type: stdout
beat: {
"hostname": "ip-10-169-1-48.ec2.internal",
"name": "ip-10-169-1-48.ec2.internal"
}
count: 1
fields: {
"environment": "NA-DEV",
"platform": "RSDMT"
}
host: ip-10-169-1-48.ec2.internal
input_type: log
message: 07:36:29,391 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
offset: 510957
source: /opt/tomcat/logs/stdout.log
tags: [
"beats_input_codec_plain_applied"
]
type: stdout

How do I need to omit above json code from alert mail. Please suggest.

2

You will probably need to ask the authors directly, I don't believe they hang out on these forums.

3

In which forum do I get help regarding elastalert. Or could you please let me know the right place to get help

4

Try their github repository?

5

Ok, thankyou

6

Just one more help.

Could you please let me know how to deal with below error message on logstash server

[root@ip-10-192-4-253 bin]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
Settings: Default pipeline workers: 4
An unexpected error occurred! {:error=>#<Errno::EADDRINUSE: Address already in use - bind - Address already in use>, :class=>"Errno::EADDRINUSE", :backtrace=>["org/jruby/ext/socket/RubyTCPServer.java:118:in initialize'", "org/jruby/RubyIO.java:853:innew'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.7/lib/lumberjack/beats/server.rb:51:in initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-beats-2.2.7/lib/logstash/inputs/beats.rb:119:inregister'",

7

Whatever your config is trying to do, it looks like there is something else already using the address/port.

8

Try use_count_query: true in your rule, it should omit the json data from events.

9

thankyou :slight_smile:
it worked partially. I mean the JSON script is omitted but the email contents are not as expected. anyways i used "alert_text_type: alert_text_only" to solve it.

11

Is it possible to send the data to Kafka Topic ? (the json message data to kafka topic)

The json data message to kafka topic using the rules configuration?