Before Heartbeat was GA, I already set up a Logstash pipeline to monitor our application's HTTP endpoint for uptime/downtime/errors/...
I would like to switch to Hearbeat now, so I can also easily monitor the system on other levels (webserver/appserver/db/...) as well. However I don't see a way to implement one feature I use in my current Logstash pipeline: I can set a field called "maintenance" to true or false, to differentiate between planned downtimes (marked blue in our dashboard) and unplanned outages (red).
My current Logstash filter (simplified):
ruby {
code => 'event.set("maintenance", File.exists?("/tmp/"+event.get("[@metadata][name]")+"_maint"))'
}
When I want to set a system to maintenance, e.g. our PROD environment, I just create a file /tmp/prod_maint (touch /tmp/prod_maint
), and remove it afterwards (rm /tmp/prod_maint
). This is easy to do with various tools, and gives me great flexibility.
I don't think I can add this with a simple processor, as those don't allow checking for files.
Does anyone have a good idea on how to do that?