I configured Filebeats to add new Logfiles in the harvestors' path and restarted the filebeat service on those instances from where the Logs were to be harvested.
On the Logstash side, I configured the Logstash-output-email plugin to send out an email once a 500 error is encountered in the message. What I didn't apprehend beforehand was that the harvesting would start from the beginning of the Logfiles. What I am trying to achieve is to harvest only for new entries and not the existing logs.
Now the real problem is that, Once the emails started coming in , they are not stopping at all eventhough I have commented the email output code in Logstash. I am assuming the emails because of the older logs have queued in and are coming out in humongous amounts. Is there any way to flush the email queue ?
On the Logstash side, I configured the Logstash-output-email plugin to send out an email once a 500 error is encountered in the message.
That's a really bad idea since there, as you've noticed, won't be any rate-limiting. If you receive thousands or even millions of bad requests you will get thousands or even millions of email messages.
Is there any way to flush the email queue ?
That depends on where the email queue resides. Is it under your control at all or are you sending the messages via your ISP or similar?
I have used the via = >"smtp" to send the emails through. I am not sure if I can control that queue .
If not that, I went through the Metric filter, and have just one question.
If I put a metric for a condition , e.g. 20 consecutive 500 errors in a minute, then how can I access the messages for those events ?