Near real time alerts for syslogs

Elastic Stack Elasticsearch
1

I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

I don't if you can do it with logstash, but from an Elasticsearch point of view, you should look at Percolator feature.
It could help you to build alerting system based on prerecorded queries.

HTH

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 29 mai 2013 à 03:10, Ryan Palamara ryan.palamara@gmail.com a écrit :

I am using Elasticsearch combined with Logstash and Kibana for collecting log data from a number of different network devices. I just set it up in the past few days and so far it has been handling the load wonderfully. I would like to setup alerts for certain events that can be taken from the logs. Things like getting an alert after a certain amount of events in a time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to do this, but was hoping that someone could help point me in the right direction.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi David,

I have the same request but, as a new user of ES, I'm interested to know why the alerting process should be moved to the Logstash layer.

I'm thinking (on a white board for now) about a logstash layer (Log->ES) followed by an ES layer (index + alerting).
I thought building requests via the percolate API to be able to centralize the alerting process instead of spanning the same "rules" on any logstash layer.

Is it so heavier in terms of CPU/IO/... on the ES layer side that you prefer move this on the Logstash layer ? Or because other reasons ?

Thanks in advance.

Christophe

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5e073c5a-7681-45db-a86d-cabefe2f4411%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hello Ryan,

I am trying to build the same type of application (device log collecting)
and I'm also very new to logstash and elasticsearch.
I'm having a hard time setting up a lab environment that can sustain the
load (2000 logs/sec, 1024ko logs) and only 60% of the logs are indexed (I
count the number of lucene doucuments).

So maybe you can give me a few tips or advices on how you tuned you
environment.

How do you start logstash? just with the script provided in the project?
Are you using the syslog plugin to listen on port 514?
How many elasticsearch nodes do you have?

I would really appreciate if you could take some time to share your
experience on this.

Thank you,

Antoine Brun

Le mercredi 29 mai 2013 03:10:12 UTC+2, Ryan Palamara a écrit :

I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d4ca2099-86d2-4071-8359-565f902f390c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Did you find out any solution yet? I am at the same situation just like
you. And one more important thing is I maybe will remove logstash from the
stack in the future. So it will be idealy to get alerting system work with
ES and Kibana (or any other plugins) only.

I found out a script running queries once period of time would work but
this is not very neat. Percolator seems to act the part of this but I am
still trying.

On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:

I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/55f5a22d-ecb6-4464-ac54-a514a8641076%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Take a look at logstash throttle plugin

Also you can use output to zabbix, nagios or email directly.

Regards,
Gabriel

On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:

I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/605da3a2-6a25-4340-a74d-1e0ec42cae99%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

7

Thanks for the tips, Gabriel! So the idea is not recommend to set alert at
ES/Kibana side but on Logstash side?

On Monday, April 6, 2015 at 8:33:21 PM UTC-4, Gabriel Rosca wrote:

Take a look at logstash throttle plugin

Throttle filter plugin | Logstash Reference [8.11] | Elastic

Also you can use output to zabbix, nagios or email directly.

Regards,
Gabriel

On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:

I am using Elasticsearch combined with Logstash and Kibana for collecting
log data from a number of different network devices. I just set it up in
the past few days and so far it has been handling the load wonderfully. I
would like to setup alerts for certain events that can be taken from the
logs. Things like getting an alert after a certain amount of events in a
time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way to
do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/08e506cb-a30c-4edf-9efc-02abb8ba90ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Well,

Depends on your needs. I like to use LS for that to take advantage of
Nagios and zabbix plugins and let the monitoring system start the
escalations procedures and log the incident.

Regards,
Gabriel

On Tue, Apr 7, 2015 at 9:53 AM, Lincoln Xiong xiong.huanglin@gmail.com
wrote:

Thanks for the tips, Gabriel! So the idea is not recommend to set alert at
ES/Kibana side but on Logstash side?

On Monday, April 6, 2015 at 8:33:21 PM UTC-4, Gabriel Rosca wrote:

Take a look at logstash throttle plugin

Throttle filter plugin | Logstash Reference [8.11] | Elastic

Also you can use output to zabbix, nagios or email directly.

Regards,
Gabriel

On Tuesday, May 28, 2013 at 9:10:12 PM UTC-4, Ryan Palamara wrote:

I am using Elasticsearch combined with Logstash and Kibana for
collecting log data from a number of different network devices. I just set
it up in the past few days and so far it has been handling the load
wonderfully. I would like to setup alerts for certain events that can be
taken from the logs. Things like getting an alert after a certain amount of
events in a time period or alerts for certain log events.

Now I am very new at this and have been searching through for some way
to do this, but was hoping that someone could help point me in the right
direction.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9l59dNQIALk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/08e506cb-a30c-4edf-9efc-02abb8ba90ed%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/08e506cb-a30c-4edf-9efc-02abb8ba90ed%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALtJ1PvguZDDYzf9%3DV%3DZC_He%3DLohuTi6Xb-hT-Ggv7py2LmvEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.