I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.
I don't if you can do it with logstash, but from an Elasticsearch point of view, you should look at Percolator feature.
It could help you to build alerting system based on prerecorded queries.
HTH
--
David
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
I am using Elasticsearch combined with Logstash and Kibana for collecting log data from a number of different network devices. I just set it up in the past few days and so far it has been handling the load wonderfully. I would like to setup alerts for certain events that can be taken from the logs. Things like getting an alert after a certain amount of events in a time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to do this, but was hoping that someone could help point me in the right direction.
I have the same request but, as a new user of ES, I'm interested to know why the alerting process should be moved to the Logstash layer.
I'm thinking (on a white board for now) about a logstash layer (Log->ES) followed by an ES layer (index + alerting).
I thought building requests via the percolate API to be able to centralize the alerting process instead of spanning the same "rules" on any logstash layer.
Is it so heavier in terms of CPU/IO/... on the ES layer side that you prefer move this on the Logstash layer ? Or because other reasons ?
I am trying to build the same type of application (device log collecting)
and I'm also very new to logstash and elasticsearch.
I'm having a hard time setting up a lab environment that can sustain the
load (2000 logs/sec, 1024ko logs) and only 60% of the logs are indexed (I
count the number of lucene doucuments).
So maybe you can give me a few tips or advices on how you tuned you
environment.
How do you start logstash? just with the script provided in the project?
Are you using the syslog plugin to listen on port 514?
How many elasticsearch nodes do you have?
I would really appreciate if you could take some time to share your
experience on this.
Thank you,
Antoine Brun
Le mercredi 29 mai 2013 03:10:12 UTC+2, Ryan Palamara a écrit :
I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.
Did you find out any solution yet? I am at the same situation just like
you. And one more important thing is I maybe will remove logstash from the
stack in the future. So it will be idealy to get alerting system work with
ES and Kibana (or any other plugins) only.
I found out a script running queries once period of time would work but
this is not very neat. Percolator seems to act the part of this but I am
still trying.
On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:
I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.
Also you can use output to zabbix, nagios or email directly.
Regards,
Gabriel
On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:
I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.
Also you can use output to zabbix, nagios or email directly.
Regards,
Gabriel
On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:
I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.
Depends on your needs. I like to use LS for that to take advantage of
Nagios and zabbix plugins and let the monitoring system start the
escalations procedures and log the incident.
Also you can use output to zabbix, nagios or email directly.
Regards,
Gabriel
On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:
I am using Elasticsearch combined with Logstash and Kibana for
collecting log data from a number of different network devices. I just set
it up in the past few days and so far it has been handling the load
wonderfully. I would like to setup alerts for certain events that can be
taken from the logs. Things like getting an alert after a certain amount of
events in a time period or alerts for certain log events.
Now I am very new at this and have been searching through for some way
to do this, but was hoping that someone could help point me in the right
direction.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
