Heartbeat monitor auto reloading does now work

tterranigma · September 11, 2020, 10:53am

According to heartbeat documentation, heartbeat supports automatically reloading the monitor files periodically. In my installation of heartbeat, I have enabled this:

heartbeat:
  config:
    monitors:
      path: /etc/heartbeat/monitors.d//*.yml
    reload:
      enabled: true
      period: 5s

However, this does not work and I have to restart the heartbeat service every time I create new monitors or change existing ones.

This is how the heartbeat process looks like:

root      9883 14.1  1.1 1146752 97348 ?       Ssl  09:55   7:40 /usr/share/heartbeat/bin/heartbeat -environment systemd -e -d elasticsearch -c /etc/heartbeat/heartbeat.yml -path.home /usr/share/heartbeat -path.config /etc/heartbeat -path.data /var/lib/heartbeat -path.logs /var/log/heartbeat

The heartbeat monitors have the following permissions:

-rw-r--r-- 1 root root 324 Sep 10 11:05 monitors.d/twmn.elasticsearch.yml

and the folder containing the monitors:

drwxr-xr-x 2 root root 102400 Sep 11 09:55 monitors.d/

Am I doing sth wrong?

shaunak · September 14, 2020, 1:25pm

Hi, what version of Heartbeat are you using?

Also, could you run Heartbeat with logging.level: debug set in your heartbeat.yml, then change a monitor or create a new one, and post the Heartbeat logs for the following 10 seconds or so?

Thanks,

Shaunak

tterranigma · September 15, 2020, 12:01pm

I am using version 7.8.1. I remember it being this way with a few previous versions as well. Here is what journalctl says. I run journalctl -f first, then edit a monitor from enabled: False to disabled: True, wait for half a minute and then restart the service manually to have it loaded. I remove many lines that simply count the events reported.


Sep 15 11:53:30 earth-kibana heartbeat[2998]: 2020-09-15T11:53:30.111Z        INFO        cfgfile/reload.go:224        Loading of config files completed.
Sep 15 11:53:30 earth-kibana heartbeat[2998]: 2020-09-15T11:53:30.241Z        DEBUG        [elasticsearch]        elasticsearch/client.go:229        PublishEvents: 50 events have been published to elasticsearch in 108.02437ms.
Sep 15 11:53:33 earth-kibana heartbeat[2998]: 2020-09-15T11:53:33.945Z        WARN        [tls]        tlscommon/tls_config.go:83        SSL/TLS verifications disabled.
Sep 15 11:53:33 earth-kibana heartbeat[2998]: 2020-09-15T11:53:33.985Z        DEBUG        [elasticsearch]        elasticsearch/client.go:229        PublishEvents: 50 events have been published to elasticsearch in 551.713238ms.
Sep 15 11:53:38 earth-kibana heartbeat[2998]: 2020-09-15T11:53:38.930Z        WARN        [tls]        tlscommon/tls_config.go:83        SSL/TLS verifications disabled.
Sep 15 11:53:38 earth-kibana heartbeat[2998]: 2020-09-15T11:53:38.994Z        DEBUG        [elasticsearch]        elasticsearch/client.go:229        PublishEvents: 50 events have been published to elasticsearch in 132.151003ms.
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.014Z        WARN        scheduler/scheduler.go:152        15 tasks have missed their schedule deadlines in the last 15s.
Sep 15 11:53:57 earth-kibana systemd[1]: Stopping Ping remote services for availability and log results to Elasticsearch or send to Logstash....
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.764Z        INFO        beater/heartbeat.go:120        Shutting down.
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.808Z        INFO        cfgfile/reload.go:227        Dynamic config reloader stopped
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.808Z        INFO        [reload]        cfgfile/list.go:118        Stopping 963 runners ...
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.863Z        INFO        [api]        api/server.go:66        Stats endpoint (192.168.1.5:5070) finished: accept tcp 192.168.1.5:5070: use of closed network connection
Sep 15 11:53:57 earth-kibana heartbeat[2998]: 2020-09-15T11:53:57.864Z        INFO        instance/beat.go:469        heartbeat stopped.
Sep 15 11:53:57 earth-kibana systemd[1]: Stopped Ping remote services for availability and log results to Elasticsearch or send to Logstash..
Sep 15 11:53:57 earth-kibana systemd[1]: Started Ping remote services for availability and log results to Elasticsearch or send to Logstash..
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.298Z        INFO        instance/beat.go:647        Home path: [/usr/share/heartbeat] Config path: [/etc/heartbeat] Data path: [/var/lib/heartbeat] Logs path: [/var/log/heartbeat]
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.308Z        INFO        instance/beat.go:655        Beat ID: 9805c7c4-6ede-5d01-92a3-2abe0d3960db
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.311Z        INFO        [api]        api/server.go:62        Starting stats endpoint
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.311Z        INFO        [seccomp]        seccomp/seccomp.go:124        Syscall filter successfully installed
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.313Z        INFO        [beat]        instance/beat.go:983        Beat info        {"system_info": {"beat": {"path": {"config": "/etc/heartbeat", "data": "/var/lib/heartbeat", "home": "/usr/share/heartbeat", "logs": "/var/log/heartbeat"}, "type": "heartbeat", "uuid": "9805c7c4-6ede-5d01-92a3-2abe0d3960db"}}}
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.314Z        INFO        [beat]        instance/beat.go:992        Build info        {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T14:22:56.000Z", "version": "7.8.1"}}}
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.314Z        INFO        [api]        api/server.go:64        Metrics endpoint listening on: 192.168.1.5:5070 (configured: 192.168.1.5)
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.314Z        INFO        [beat]        instance/beat.go:995        Go runtime info        {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.10"}}}
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.319Z        INFO        [beat]        instance/beat.go:999        Host info        {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-09-11T20:45:22Z","containerized":false,"name":"earth-kibana","ip":["127.0.0.1/8","::1/128","192.168.1.5/32","fe80::4001:c0ff:fea8:105/64"],"kernel_version":"5.0.0-1028-gcp","mac":["42:01:c0:a8:01:05"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"18.04.3 LTS (Bionic Beaver)","major":18,"minor":4,"patch":3,"codename":"bionic"},"timezone":"UTC","timezone_offset_sec":0,"id":"e309c5fbad3714e4eb72cfbf36c8adaa"}}}
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.320Z        INFO        [beat]        instance/beat.go:1028        Process info        {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/", "exe": "/usr/share/heartbeat/bin/heartbeat", "name": "heartbeat", "pid": 3068, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-09-15T11:53:57.570Z"}}}
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.320Z        INFO        instance/beat.go:310        Setup Beat: heartbeat; Version: 7.8.1
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.320Z        INFO        [index-management]        idxmgmt/std.go:184        Set output.elasticsearch.index to 'heartbeat-7.8.1' as ILM is enabled.
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        INFO        eslegclient/connection.go:99        elasticsearch url: https://35.206.159.154:9200
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        WARN        [tls]        tlscommon/tls_config.go:83        SSL/TLS verifications disabled.
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        INFO        eslegclient/connection.go:99        elasticsearch url: https://35.210.163.90:9200
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        WARN        [tls]        tlscommon/tls_config.go:83        SSL/TLS verifications disabled.
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        INFO        eslegclient/connection.go:99        elasticsearch url: https://35.210.170.219:9200
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.355Z        WARN        [tls]        tlscommon/tls_config.go:83        SSL/TLS verifications disabled.
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.356Z        INFO        [publisher]        pipeline/module.go:113        Beat name: earth-kibana-heartbeat
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.356Z        INFO        instance/beat.go:463        heartbeat start running.
Sep 15 11:53:58 earth-kibana heartbeat[3068]: 2020-09-15T11:53:58.356Z        INFO        beater/heartbeat.go:82        heartbeat is running! Hit CTRL-C to stop it.
Sep 15 11:54:00 earth-kibana heartbeat[3068]: 2020-09-15T11:54:00.236Z        INFO        cfgfile/reload.go:164        Config reloader started
Sep 15 11:54:00 earth-kibana heartbeat[3068]: 2020-09-15T11:54:00.466Z        ERROR        [reload]        cfgfile/list.go:96        Error creating runner from config: Monitor not loaded, plugin is disabled
Sep 15 11:54:00 earth-kibana heartbeat[3068]: 2020-09-15T11:54:00.468Z        ERROR        [reload]        cfgfile/list.go:96        Error creating runner from config: Monitor not loaded, plugin is disabled
Sep 15 11:54:00 earth-kibana heartbeat[3068]: 2020-09-15T11:54:00.469Z        ERROR        [reload]        cfgfile/list.go:96        Error creating runner from config: Monitor not loaded, plugin is disabled
(END)

shaunak · September 15, 2020, 12:37pm

Looking at the last three lines of the logs it looks like, for some reason, Heartbeat doesn't like the monitor configurations. Would you mind posting those here (with any sensitive information redacted)?

Thanks,

Shaunak

stephenb · September 16, 2020, 1:22am

Did notice you have an extra / in your path probably not the issue but it is extraneous

system · October 14, 2020, 3:22am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.