# Looking to build a logging solution with threshold alerting **URL:** \ **Category:** Elasticsearch\ **Created:** [July 2, 2014, 12:53pm UTC](https://discuss.elastic.co/t/looking-to-build-a-logging-solution-with-threshold-alerting/18441 "2014-07-02T12:53:55Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Joshua\_Hall](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshua_hall/32/1448_2.png) [@Joshua\_Hall](https://discuss.elastic.co/u/Joshua_Hall)\ **Post date:** [July 2, 2014, 12:53pm UTC](https://discuss.elastic.co/t/looking-to-build-a-logging-solution-with-threshold-alerting/18441/1 "2014-07-02T12:53:55Z")
I am looking to build a logging solution and wanted to make sure that I am not missing any key components. The logs that I have are currently stored in a database which there is limited access due to locking risks from bad queries. My plan is to have the dba's write the logs from the database tables to a file on a set interval then have logstash pick up the logs and write it to elastic search. Then for viewing/searching the logs I will be using kibana. Everything up to this point I have been able to make a proof of concept for but the other request was to have alerting. I have spent some time looking at this and the general response seems to be to use percolation, but that seems to only make sense if you want to send an alert if you receive a single error that matches a query and from what I have seen there is no way to a threshold alerting system using percolation. My thought to solve the threshold alerting is to create a simple web UI that allows the user to enter in a query to search for, a threshold, a time frame, and emails to send the alert to that would get stored in elastic search. Then an app (Running as a windows service or cron job) that pulls the alerts and then runs the queries and checks the time-frame and threshold (Would run on some interval). If the count surpasses the threshold then it would send an email to values stored in the email addresses. I know that SPM seems to cover this and move but we are currently looking to see if we can do this without buying another product. Is this the correct approach to take or should I be looking at doing something else? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 2, 2014, 11:43pm UTC](https://discuss.elastic.co/t/looking-to-build-a-logging-solution-with-threshold-alerting/18441/2 "2014-07-02T23:43:08Z")
There was another thread on this very recently, and some people are using riemann for this. Take a look in the archives and you can probably find some useful info. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 2 July 2014 22:53, Joshua Hall [joshuadeanhall@gmail.com](mailto:joshuadeanhall@gmail.com) wrote: > I am looking to build a logging solution and wanted to make sure that I am > not missing any key components. > > The logs that I have are currently stored in a database which there is > limited access due to locking risks from bad queries. > > My plan is to have the dba's write the logs from the database tables to a > file on a set interval then have logstash pick up the logs and write it to > Elasticsearch. Then for viewing/searching the logs I will be using > kibana. Everything up to this point I have been able to make a proof of > concept for but the other request was to have alerting. > > I have spent some time looking at this and the general response seems to > be to use percolation, but that seems to only make sense if you want to > send an alert if you receive a single error that matches a query and from > what I have seen there is no way to a threshold alerting system using > percolation. > > My thought to solve the threshold alerting is to create a simple web UI > that allows the user to enter in a query to search for, a threshold, a time > frame, and emails to send the alert to that would get stored in elastic > search. Then an app (Running as a windows service or cron job) that pulls > the alerts and then runs the queries and checks the time-frame and > threshold (Would run on some interval). If the count surpasses the > threshold then it would send an email to values stored in the email > addresses. > > I know that SPM seems to cover this and move but we are currently looking > to see if we can do this without buying another product. > > Is this the correct approach to take or should I be looking at doing > something else? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ce1cb3cc-e974-4b3b-8568-a2afaaae6c00%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Z2f%3DD9H1LfWX98oTNNJia2R1-NEwkpiEtZ63FiKrOmGA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Z2f%3DD9H1LfWX98oTNNJia2R1-NEwkpiEtZ63FiKrOmGA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:18am UTC](https://discuss.elastic.co/t/looking-to-build-a-logging-solution-with-threshold-alerting/18441/3 "2017-07-06T01:18:15Z")