Setting up alerts


(Joshua Hall) #1

I am working on a demo using Elasticsearch, Logstash, Kibana and one of the
key features that I am looking for is the ability to setup alerts to send
out emails.

Specifically I want to setup an alert to be emailed when a log is recorded
with "severity=error" and "category=category1" occurs more than 20 times in
15 minutes.

I am limited to a windows only solution so that may limit some things.

I have considered the following approaches

  1. An app/service that simply does a search to get the number of results
    every minute or so
  2. Doing something with logstash
  3. Doing something with Percolator (Logstash or an app)

I really don't understand the percolator so I am not sure if it is even a
viable approach for what doing this.

Is there any guide / general guidance for doing something similar to this?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(vineeth mohan-2) #2

Hello Joshua ,

Percolater is the usual choice here.
You can register queries against poercolator and when you index some feed ,
the peorcolater informs you that such a search query matches against your
index.
This information is passed when indexing.

With logstash , I am not sure how this can be done.

Thanks
Vineeth

On Sat, May 31, 2014 at 8:31 AM, Joshua Hall joshuadeanhall@gmail.com
wrote:

I am working on a demo using Elasticsearch, Logstash, Kibana and one of
the key features that I am looking for is the ability to setup alerts to
send out emails.

Specifically I want to setup an alert to be emailed when a log is recorded
with "severity=error" and "category=category1" occurs more than 20 times in
15 minutes.

I am limited to a windows only solution so that may limit some things.

I have considered the following approaches

  1. An app/service that simply does a search to get the number of
    results every minute or so
  2. Doing something with logstash
  3. Doing something with Percolator (Logstash or an app)

I really don't understand the percolator so I am not sure if it is even a
viable approach for what doing this.

Is there any guide / general guidance for doing something similar to this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5%3D3dP5FcLHDevgZeOmYsAhvReUAQPLm1Nh_651a0Kjt1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Joshua Hall) #3

Vinneth thanks for your response.

I guess I don't understand how the percolater works. I get that you store
a query, In my case I think I would want to store a query that looks at the
fields I want for a specific value over the last timeperiod.

Does the percolater actually attach to an index and run against all new
documents added to the index?

When I look at the documentation and see this I get very confused

curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{
"doc" : {
"message" : "A new bonsai tree in the office"
}
}'

To me this looks like asking the percolator if this message matches a
percolator, but what I want to know is the count of the number of documents
added to an index in the last 15 minutes matches a percolator.

I would have expected to see something more similiar to

curl -XGET 'localhost:9200/index/type/_percolate/count'

Hopefully this makes sense as I am very confused about how this works,
looks to me like it is just a simple does this document match any of the
predefined queries and what I think I am after is does any documents in
this index match this predefined query.

On Saturday, May 31, 2014 2:18:33 AM UTC-4, vineeth mohan wrote:

Hello Joshua ,

Percolater is the usual choice here.
You can register queries against poercolator and when you index some feed
, the peorcolater informs you that such a search query matches against your
index.
This information is passed when indexing.

With logstash , I am not sure how this can be done.

Thanks
Vineeth

On Sat, May 31, 2014 at 8:31 AM, Joshua Hall <joshuad...@gmail.com
<javascript:>> wrote:

I am working on a demo using Elasticsearch, Logstash, Kibana and one of
the key features that I am looking for is the ability to setup alerts to
send out emails.

Specifically I want to setup an alert to be emailed when a log is
recorded with "severity=error" and "category=category1" occurs more than 20
times in 15 minutes.

I am limited to a windows only solution so that may limit some things.

I have considered the following approaches

  1. An app/service that simply does a search to get the number of
    results every minute or so
  2. Doing something with logstash
  3. Doing something with Percolator (Logstash or an app)

I really don't understand the percolator so I am not sure if it is even a
viable approach for what doing this.

Is there any guide / general guidance for doing something similar to this?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/98e8ec88-7ef9-45e6-875b-e72f72a4731a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #4

With logstash, you can use either the email or pagerduty outputs. You can
setup a conditional for your specific log entries.

http://logstash.net/docs/1.4.1/outputs/email
http://logstash.net/docs/1.4.1/outputs/pagerduty

--
Ivan

On Sat, May 31, 2014 at 10:11 AM, Joshua Hall joshuadeanhall@gmail.com
wrote:

Vinneth thanks for your response.

I guess I don't understand how the percolater works. I get that you store
a query, In my case I think I would want to store a query that looks at the
fields I want for a specific value over the last timeperiod.

Does the percolater actually attach to an index and run against all new
documents added to the index?

When I look at the documentation and see this I get very confused

curl -XGET 'localhost:9200/my-index/message/_percolate' -d '{
"doc" : {
"message" : "A new bonsai tree in the office"
}
}'

To me this looks like asking the percolator if this message matches a
percolator, but what I want to know is the count of the number of documents
added to an index in the last 15 minutes matches a percolator.

I would have expected to see something more similiar to

curl -XGET 'localhost:9200/index/type/_percolate/count'

Hopefully this makes sense as I am very confused about how this works,
looks to me like it is just a simple does this document match any of the
predefined queries and what I think I am after is does any documents in
this index match this predefined query.

On Saturday, May 31, 2014 2:18:33 AM UTC-4, vineeth mohan wrote:

Hello Joshua ,

Percolater is the usual choice here.
You can register queries against poercolator and when you index some feed
, the peorcolater informs you that such a search query matches against your
index.
This information is passed when indexing.

With logstash , I am not sure how this can be done.

Thanks
Vineeth

On Sat, May 31, 2014 at 8:31 AM, Joshua Hall joshuad...@gmail.com
wrote:

I am working on a demo using Elasticsearch, Logstash, Kibana and one of
the key features that I am looking for is the ability to setup alerts to
send out emails.

Specifically I want to setup an alert to be emailed when a log is
recorded with "severity=error" and "category=category1" occurs more than 20
times in 15 minutes.

I am limited to a windows only solution so that may limit some things.

I have considered the following approaches

  1. An app/service that simply does a search to get the number of
    results every minute or so
  2. Doing something with logstash
  3. Doing something with Percolator (Logstash or an app)

I really don't understand the percolator so I am not sure if it is even
a viable approach for what doing this.

Is there any guide / general guidance for doing something similar to
this?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/82b6748d-0de9-4de0-92f3-65fcaeb69d3d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/98e8ec88-7ef9-45e6-875b-e72f72a4731a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/98e8ec88-7ef9-45e6-875b-e72f72a4731a%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDCoV_vpb9uT2F_7Xzg%2ByMfgHx6qpkU6vqC6%2BAWcbL-ag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5