I am looking to build a logging solution and wanted to make sure that I am
not missing any key components.
The logs that I have are currently stored in a database which there is
limited access due to locking risks from bad queries.
My plan is to have the dba's write the logs from the database tables to a
file on a set interval then have logstash pick up the logs and write it to
elastic search. Then for viewing/searching the logs I will be using
kibana. Everything up to this point I have been able to make a proof of
concept for but the other request was to have alerting.
I have spent some time looking at this and the general response seems to be
to use percolation, but that seems to only make sense if you want to send
an alert if you receive a single error that matches a query and from what I
have seen there is no way to a threshold alerting system using percolation.
My thought to solve the threshold alerting is to create a simple web UI
that allows the user to enter in a query to search for, a threshold, a time
frame, and emails to send the alert to that would get stored in elastic
search. Then an app (Running as a windows service or cron job) that pulls
the alerts and then runs the queries and checks the time-frame and
threshold (Would run on some interval). If the count surpasses the
threshold then it would send an email to values stored in the email
addresses.
I know that SPM seems to cover this and move but we are currently looking
to see if we can do this without buying another product.
Is this the correct approach to take or should I be looking at doing
something else?
There was another thread on this very recently, and some people are using
riemann for this.
Take a look in the archives and you can probably find some useful info.
I am looking to build a logging solution and wanted to make sure that I am
not missing any key components.
The logs that I have are currently stored in a database which there is
limited access due to locking risks from bad queries.
My plan is to have the dba's write the logs from the database tables to a
file on a set interval then have logstash pick up the logs and write it to
Elasticsearch. Then for viewing/searching the logs I will be using
kibana. Everything up to this point I have been able to make a proof of
concept for but the other request was to have alerting.
I have spent some time looking at this and the general response seems to
be to use percolation, but that seems to only make sense if you want to
send an alert if you receive a single error that matches a query and from
what I have seen there is no way to a threshold alerting system using
percolation.
My thought to solve the threshold alerting is to create a simple web UI
that allows the user to enter in a query to search for, a threshold, a time
frame, and emails to send the alert to that would get stored in elastic
search. Then an app (Running as a windows service or cron job) that pulls
the alerts and then runs the queries and checks the time-frame and
threshold (Would run on some interval). If the count surpasses the
threshold then it would send an email to values stored in the email
addresses.
I know that SPM seems to cover this and move but we are currently looking
to see if we can do this without buying another product.
Is this the correct approach to take or should I be looking at doing
something else?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.