Monitor inactivity in log files

Elastic Stack Logstash
1

I have certain log files being monitored through ELK stack using filbebeat to send files to log stash. Suppose if the server is down and the logs are no longer being written, i want to monitor the same. Is there some way to do the same using log stash , such that if there are no events being generated by log stash say for last 1 hour, then generate an alert

2

Any help appreciated. Can logstash generate a dummy event if it doesnt receive any new records from filebeat after a period of time?

3

Have you looked at the heartbeat input plugin?

4

Logstash can alert and take action on the presence of events but not on the absence of them. I'd use Lovebeat for this.

5

So If i use love beat with filbebeat and log stash, filbebeat will have to send two outputs right? One to love beat and another to log stash

6

I don't think you can connect Filebeat to Lovebeat, but you can definitely hook up Logstash to Lovebeat.

7

Ok cool..thanks..let me try it out

8

I have love beat running in localhost 8127 port.
Modified log stash output conf file as:
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
statsd {
host => "localhost"
port => "8127"
count => {
"http.bytes" => "%{bytes}"
}
}
}

Output is being sent to elastic search running at 9200. However love beat isn't receiving any information on port 8127. Any idea where is the issue

9

Any help?

10

On manually sending data to love beat, its receiving it:
echo "logstash.health:1|c" | nc -4u -w0 localhost 8127

However the statsd output send by log stash is not reaching love beat

11

Any help?

Jeez, have patience.

On manually sending data to love beat, its receiving it:
echo "logstash.health:1|c" | nc -4u -w0 localhost 8127

However the statsd output send by log stash is not reaching love beat

Any clues in the Logstash logs? What if you crank if the log level by starting it with --debug?

12

Sorry about that.
I managed to get data to love beat by changing host to "127.0.0.1"
For a service, log stash.monitor, i assigned timeout of 10s and it changes state after 10s to error as expected.
2016/05/23 14:10:45 INFO SERVICE 'logstash.monitor', state ok -> error

However, the alarm i configured against the service pattern isn't getting triggered.Any clue why?
[[alarms]]
name = "example"
pattern = "logstash.*"
[[alarms.alerts]]
script = "/home/user/test_alarm.sh"

13

I have no idea OTOH. Is there nothing at all in the Lovebeat logs about the script execution? Does it work if you configure an email alert instead of a script alert?

14

The syntax you specified looks good for version 1.0.0-rc5. Please check that your script is executable (chmod a+x test_alarm.sh) and that it contains a valid interpreter as its first line (#!/bin/bash or similar). Also check that lovebeat doesn't show any errors on startup when parsing your configuration file.

The lovebeat configuration syntax is changing a bit as it's getting ready for version 1.0 (and then it really will be final), so already in 1.0.0-rc6 (due tomorrow) it will be like this instead:

[[alarms]]
name = "test-$name"
pattern = "test.$name.*"
alerts = ["to-slack", "run-my-script"]

[alerts.to-slack]
slack_channel = "#ops"

[alerts.run-my-script]
script = "/path/to/script.sh"

If you still have any issues, please open up an issue on the lovebeat github page, thanks!

15

Thanks victor