I have Logstash (receiver) -> redis buffer -> Logstash (indexer) -> ES cluster
I created a single heartbeat input on each Logstash instance and output to file, for example on the receiver:
input {
heartbeat {
add_field => { "scope" => "receiver" }
message => "epoch"
interval => 120
type => "heartbeat"
}
}
Each plugin has a scope field used in output {} to decide which file to log to.
output {
if [type] == "heartbeat" and [scope] == "receiver" {
file {
path => "/var/log/logstash/heartbeat-receiver.log"
}
} else {
....
That worked fine. I then added a second one on the receiver with the same interval with the intention that it would be logged by the indexer to show throughput across the entire system.
With all three heartbeats set to 120 seconds, I only saw logs for the receiver and indexer and not the second one that traverses the whole system. When I set the intervals to 53 and 127 (prime numbers to avoid collisions) I got all three logs. However, I expect I'll lose one every 53 x 127 seconds.
#apt-cache policy logstash
logstash:
Installed: 1:2.3.4-1
Candidate: 1:2.3.4-1