We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs. Eg
When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric
filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this. Thanks
The solutions I've seen for things like this in ELK usually are on the
lines of using logstash to reparse the logs in ES and use some output
(e-mail, nagios, Zabbix) to do the alerting.
For now I've stick with using OSSEC (www.ossec.net) to do my alerting and
"just" use ELK for log analysis.
On Monday, June 23, 2014 5:50:22 AM UTC-3, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric
filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
@Antonio: I use email output for a particular pattern in a log. But for
example, when a particular logs comes more than 5 times, for alerting for
this a state needs to be maintained which is not there in logstash.
I don't know about OSSEC. But how to use it to achieve the above? Presently
logstash reads logs, ES stores it and kibana presents it. How OSSEC fits
here?
On Monday, 23 June 2014 15:50:36 UTC+5:30, Antonio Augusto Santos wrote:
The solutions I've seen for things like this in ELK usually are on the
lines of using logstash to reparse the logs in ES and use some output
(e-mail, nagios, Zabbix) to do the alerting.
For now I've stick with using OSSEC (www.ossec.net) to do my alerting and
"just" use ELK for log analysis.
On Monday, June 23, 2014 5:50:22 AM UTC-3, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
I haven't done it yet.. but my plan is to simply do REST searches.. and if
I get a lot of alerts and want to check often.. I'd switch to setting up
percolators.
OSSEC is a HIDS (Host Intrusion Detection SYstem), its objective is to
parse logs, check the logs against rules and send alerts. It has a vast
amount of rules already defined, so when it starts checking your logs it
will start firing alerts.
In my architecture OSSEC fits outside the ELK stack. It parses the logs (as
Logstash) but write alerts to files and send alerts by email.
I've another logstash instance that reads the alerts from OSSEC, so I can
have a nice Dashboard for them on Kibana.
On Monday, June 23, 2014 7:49:03 AM UTC-3, Siddharth Trikha wrote:
@Antonio: I use email output for a particular pattern in a log. But for
example, when a particular logs comes more than 5 times, for alerting for
this a state needs to be maintained which is not there in logstash.
I don't know about OSSEC. But how to use it to achieve the above?
Presently logstash reads logs, ES stores it and kibana presents it. How
OSSEC fits here?
On Monday, 23 June 2014 15:50:36 UTC+5:30, Antonio Augusto Santos wrote:
The solutions I've seen for things like this in ELK usually are on the
lines of using logstash to reparse the logs in ES and use some output
(e-mail, nagios, Zabbix) to do the alerting.
For now I've stick with using OSSEC (www.ossec.net) to do my alerting
and "just" use ELK for log analysis.
On Monday, June 23, 2014 5:50:22 AM UTC-3, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to
analyze our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
So inputing log files to logstash-parse-store-view them and separately
doing this for alerts:ossec-parse-alert, will this not create redudancy??
Does OSSEC has rules for threshold crossing?? Any suggestions which fits my
scenario best?
On Monday, 23 June 2014 16:55:54 UTC+5:30, Antonio Augusto Santos wrote:
OSSEC is a HIDS (Host Intrusion Detection SYstem), its objective is to
parse logs, check the logs against rules and send alerts. It has a vast
amount of rules already defined, so when it starts checking your logs it
will start firing alerts.
In my architecture OSSEC fits outside the ELK stack. It parses the logs
(as Logstash) but write alerts to files and send alerts by email.
I've another logstash instance that reads the alerts from OSSEC, so I can
have a nice Dashboard for them on Kibana.
On Monday, June 23, 2014 7:49:03 AM UTC-3, Siddharth Trikha wrote:
@Antonio: I use email output for a particular pattern in a log. But for
example, when a particular logs comes more than 5 times, for alerting for
this a state needs to be maintained which is not there in logstash.
I don't know about OSSEC. But how to use it to achieve the above?
Presently logstash reads logs, ES stores it and kibana presents it. How
OSSEC fits here?
On Monday, 23 June 2014 15:50:36 UTC+5:30, Antonio Augusto Santos wrote:
The solutions I've seen for things like this in ELK usually are on the
lines of using logstash to reparse the logs in ES and use some output
(e-mail, nagios, Zabbix) to do the alerting.
For now I've stick with using OSSEC (www.ossec.net) to do my alerting
and "just" use ELK for log analysis.
On Monday, June 23, 2014 5:50:22 AM UTC-3, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to
analyze our logs. So far, so good.
But now we want notification generation on some particular kind of
logs. Eg When a login failed logs comes more than 5 times (threshold
crossed) an email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
Yes, it adds redundancy, but, for now, I think its the best option.
OSSEC has a very flexible analysing system, and it should fit your needs
without much trouble.
On Monday, June 23, 2014 8:42:46 AM UTC-3, Siddharth Trikha wrote:
So inputing log files to logstash-parse-store-view them and separately
doing this for alerts:ossec-parse-alert, will this not create redudancy??
Does OSSEC has rules for threshold crossing?? Any suggestions which fits
my scenario best?
On Monday, 23 June 2014 16:55:54 UTC+5:30, Antonio Augusto Santos wrote:
OSSEC is a HIDS (Host Intrusion Detection SYstem), its objective is to
parse logs, check the logs against rules and send alerts. It has a vast
amount of rules already defined, so when it starts checking your logs it
will start firing alerts.
In my architecture OSSEC fits outside the ELK stack. It parses the logs
(as Logstash) but write alerts to files and send alerts by email.
I've another logstash instance that reads the alerts from OSSEC, so I can
have a nice Dashboard for them on Kibana.
On Monday, June 23, 2014 7:49:03 AM UTC-3, Siddharth Trikha wrote:
@Antonio: I use email output for a particular pattern in a log. But for
example, when a particular logs comes more than 5 times, for alerting for
this a state needs to be maintained which is not there in logstash.
I don't know about OSSEC. But how to use it to achieve the above?
Presently logstash reads logs, ES stores it and kibana presents it. How
OSSEC fits here?
On Monday, 23 June 2014 15:50:36 UTC+5:30, Antonio Augusto Santos wrote:
The solutions I've seen for things like this in ELK usually are on the
lines of using logstash to reparse the logs in ES and use some output
(e-mail, nagios, Zabbix) to do the alerting.
For now I've stick with using OSSEC (www.ossec.net) to do my alerting
and "just" use ELK for log analysis.
On Monday, June 23, 2014 5:50:22 AM UTC-3, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to
analyze our logs. So far, so good.
But now we want notification generation on some particular kind of
logs. Eg When a login failed logs comes more than 5 times (threshold
crossed) an email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
We use Nagios for alerting. I originally was using the nsca output plugin
for logstash, but found that it took close to a second to execute the
command line nsca client, and if we got flooded with alert messages,
logstash would fall behind. I've since switched to use the http output and
send json to the nagios-api server (GitHub - zorkian/nagios-api: A REST-like, JSON interface to Nagios).
That seems to scale a lot better.
We do also have metrics sent from logstash to statsd/graphite, but mostly
so I can see message rates.
mike
On Monday, June 23, 2014 4:50:22 AM UTC-4, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric
filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
We have and use SPM http://sematext.com/spm/ for all our metrics (ES,
Kafka, Apache, MySQL, Hadoop, everything) and we feed our logs to Logsene http://sematext.com/logsene/ (it has a Kibana UI and a "native" UI). SPM
has alerting and anomaly detection, so we use that to get out of bed early
(nah, not really), but we currently lack alerting in Logsene (i.e. alerting
on numerical data in logs or on patterns). Since Logsene has Kibana UI and
can be fed via Logstash and has an Elasticsearch API and backend, that's
the closest we've gotten to ELK+Alerts.
On Wednesday, June 25, 2014 11:18:01 AM UTC-4, Michael Hart wrote:
We use Nagios for alerting. I originally was using the nsca output plugin
for logstash, but found that it took close to a second to execute the
command line nsca client, and if we got flooded with alert messages,
logstash would fall behind. I've since switched to use the http output and
send json to the nagios-api server (GitHub - zorkian/nagios-api: A REST-like, JSON interface to Nagios).
That seems to scale a lot better.
We do also have metrics sent from logstash to statsd/graphite, but mostly
so I can see message rates.
mike
On Monday, June 23, 2014 4:50:22 AM UTC-4, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
Nagios recently released a new Log Analysis http://www.nagios.com/products/nagios-log-server product called Nagios
Log Server which is built using the ELK stack and provides alerting,
authentication, GUI logstash configuration for the whole cluster and a ton
more.
Alerts based on any query can be sent directly to Nagios, Email, SNMP
traps, custom script execution and more...
Scott
On Monday, June 23, 2014 3:50:22 AM UTC-5, Siddharth Trikha wrote:
We are using the ELK stack (logstash, elasticsearch, kibana) to analyze
our logs. So far, so good.
But now we want notification generation on some particular kind of logs.
Eg When a login failed logs comes more than 5 times (threshold crossed) an
email to be sent to the sysadmin.
I looked up online and heard about statsd, riemann, nagios, metric
filter (logstash) to achieve our requirement.
Can anyone suggest which fits best with ELK stack?? I am new to this.
Thanks
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.