Heartbeat.config.monitors option not working on 6.5.4?

[leandrojmp (Leandro Pereira)]

Hello,

I'm trying to use heartbeat to monitor the uptime of some devices and looking into the documentation I saw that is possible to have your monitors in a separated file instead of having them in the main heartbeat.yml file.

I created the file uptime.yml inside the folder /etc/heartbeat/monitors.d/ and put my monitors there, each monitor is like the following, and I have thousands of them.

- type: icmp
  schedule: "* */5 * * * * *"
  hosts: ["host-0001"]

On my heartbeat.yml I used the following configuration for the monitoring:

heartbeat.config.monitors:
  path: /etc/heartbeat/monitors.d/*.yml
  reload.enabled: true
  reload.period: 3600s

But this does not work, the devices are not being monitored.

If I put the monitors inside heartbeat.yml and use heartbeat.monitors: as show below, everything works.

heartbeat.monitors:
- type: icmp
  schedule: "* */5 * * * * *"
  hosts: ["host-0001"]
- type: icmp
  schedule: "* */5 * * * * *"
  hosts: ["host-0002"]
- type: icmp
  schedule: "* */5 * * * * *"
  hosts: ["host-NNNN"]

Am I missing something? I just following the documentation provided here https://www.elastic.co/guide/en/beats/heartbeat/6.5/configuration-heartbeat-options.html

[Andrew_Cholakian1 (Andrew Cholakian)]

Are you sure you meant to use * */5 * * * * * as the cron syntax? That will check the monitors once every 5 hours. So, it may take up to 5 hours to see any data. It's probably easier to use Heartbeat's @every syntax, where you could write @every 5m

[leandrojmp (Leandro Pereira)]

It's every 5 minutes, not every 5 hours.

It is working when the monitors are on the heartbeat.yml, but it does not work when I configure it to use the heartbeat.config.monitors and put the monitors on another file.

[leandrojmp (Leandro Pereira)]

Updated heartbeat to 6.6 but had the same problem, if the monitors are on separated files they do not work.

Tested using the path as a reference to path.config, but no change, using quotes or not also made no difference.

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 3600s

[Andrew_Cholakian1 (Andrew Cholakian)]

I've confirmed that 6.6 does work correctly with the given config. The cron config is however incorrect, the first position is minutes, the second position is hours.

Can you give it a shot with a schedule of @every 5m or "*/1 * * * * * *"

[leandrojmp (Leandro Pereira)]

I will try to change it to @every 5m when I get back to work to see if it solves the problem, but I dont think that the cron expression is wrong, the documentation even has an example where the first position is seconds.

And as I showed on a previous post, the monitoring is working when I use the monitors on heartbeat.yml with heartbeat.monitors, but it does not work if I use an external file and use heartbeat.config.monitors, that is what I'm trying to understand what is missing.

[leandrojmp (Leandro Pereira)]

Hello,

I made some tests today and I think I've found the problem.

When using heartbeat.config.monitors heartbeat is only loading the configuration after the reload.period has elapsed.

On my configuration I was using reload.period equal as 3600s, when I changed the reload.period to 30s I saw that the monitors set with the schedule of @every 5m started appearing 5m+30s after heartbeat was started.

As you can see in the prints below, I've started heartbeat at 12:45:03, so the beats should be triggered on 12:50:03 but they were only triggered at 12:50:33.

Heartbeat started time print.

First beat triggered 5m + 30s after heartbeat start time and after that, each 5 minutes.

For another test I've changed my reload.period to 45s and could confirm that the configuration is only loadaded after the reload.period time has elapsed.

Heartbeat started at 13:20:00

First beat after schedule time + reload.period