Inside logstash, what is the official mechanism to know about any critical errors that happen with plugins / inputs / filters during the production run?
For example, I am using http_poller input and elastic output. Say, if for some reason the http_poller is unable to read the input properly from the url or elastic unable to save the output - is there anyway I can configure logstash to send me alert mails ?
Generating debug / stdout errors to console would not help in backend systems case - right?
There's no "official" method here and Logstash has no built-in alerting mechanism. What you can do is use the heartbeat input plugin to periodically generate stimuli (see the How to check Logstash's pulse blog post from May) and feed those messages and the http_poller messages to some other system that notices if messages stop coming. I'd recommend Lovebeat, which offers self-learning detection of changes in heartbeat frequency and supports both Graphite and statsd input (both of which are also supported by Logstash).
The HTTP poller will emit a special failure event in this case. You can detect it and act on it in your logstash pipeline, or use watcher to check for those events in your elasticsearch database (which is a little easier).
Check out Elastalert, you can configure it to query elasticsearch every X and alert if something specific happens or something unspecific like there is a flatline.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.