Hello, I've got following problem.
we've got Logstash instance running on server which sends output to remote server at the customer. Customer insisted on Splunk so we are sending parsed events from Logstash to Splunk. Everything is ok but during testing Splunk server was sometimes down. I've configured persistent queue on Logstash side in case of Splunk failure, however, I would like to know if there is any way how could Logstash tell us if output is not available.
We set up Logstash as a service (it's Windows server) so we don't have opened Command Prompt with running Logstash so we can't see if output is not reachable.
I've noticed there is Hearbeat which checks for availability of output. However, I don't see any benefit of it. If I send Heartbeat events to Logstash, I can't see them if Splunk is not reachable - Logstash simply pause processing.
If I send Heartbeat messages to console it's the same if I would run Logstash within Command Prompt - I would see Splunk is not reachable directly in Logstash, I don't need Heartbeat for it.
I'm very new in ELK so I'm sorry if I missed something. The point is:
Logstash is running correctly in the background and suddenly Splunk is down. Is there some way how to notice user about situation that Splunk is not reachable anymore?
To be more specific - can Logstash create syslog message in case output host is not reachable? We don't want to redirect events to somewhere else, the events are buffered and that's ok. We just want to create syslog message which will raise alarm in our monitor system.
Thank you