Heartbeat dynamic config reloader stopped

Elastic Stack Beats
1

at a random time the heartbeat is shutting down with INFO messages :
INFO cfgfile/reload.go:229 Dynamic config reloader stopped
INFO [reload] cfgfile/list.go:118 Stopping 9 runners ...
INFO [monitoring] log/log.go:154 Uptime: 2h22m49.087954637s
INFO [monitoring] log/log.go:131 Stopping metrics logging.
INFO instance/beat.go:432 heartbeat stopped.

--------- Paths are not the default ------------
path.home: /opt/beats/7.4.0/heartbeat/
path.config: /opt/beats/7.4.0/heartbeat/
path.data: /opt/beats/7.4.0/heartbeat/data/
path.logs: /logs/beats/heartbeat/

We do not know why and how to investigate further .

Thank you,

2

Can you add more logs around that for context? Is it possible that something is sending heartbeat a SIGTERM to shutdown? It looks like a clean shutdown to me.

3

The behavior is like it's having a consecutive number of loops and then it is closing the services . There is no error logged in heartbeat.log
2019-10-15T13:28:06.020+0300 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=2415, end-seq=2415
2019-10-15T13:28:06.020+0300 DEBUG [publisher] memqueue/eventloop.go:535 broker ACK events: count=1, start-seq=2415, end-seq=2415
2019-10-15T13:28:06.020+0300 DEBUG [publisher] memqueue/ackloop.go:128 ackloop: return ack to broker loop:5
2019-10-15T13:28:06.020+0300 DEBUG [publisher] memqueue/ackloop.go:131 ackloop: done send ack
2019-10-15T13:28:07.829+0300 DEBUG [service] service/service.go:55 Received sighup, stopping
2019-10-15T13:28:07.829+0300 INFO beater/heartbeat.go:118 Shutting down.
2019-10-15T13:28:07.829+0300 INFO cfgfile/reload.go:229 Dynamic config reloader stopped
2019-10-15T13:28:07.829+0300 INFO [reload] cfgfile/list.go:118 Stopping 9 runners ...
2019-10-15T13:28:07.829+0300 DEBUG [scheduler] scheduler/scheduler.go:326 done
2019-10-15T13:28:07.829+0300 DEBUG [scheduler] scheduler/scheduler.go:327 Scheduler stopped.
2019-10-15T13:28:07.829+0300 DEBUG [reload] cfgfile/list.go:129 Stopping runner: Monitor<pluginName: ElasticCluster_node_3, enabled: false>
2019-10-15T13:28:07.829+0300 DEBUG [reload] cfgfile/list.go:129 Stopping runner: Monitor<pluginName: PostgreSQL_OpenNMS, enabled: false>
2019-10-15T13:28:07.829+0300 DEBUG [reload] cfgfile/list.go:129 Stopping runner: Monitor<pluginName: Kibana_Production, enabled: false>

Config details :
heartbeat.config.monitors:
path: /opt/beats/7.4.0/heartbeat/monitors.d/*.yml
reload.enabled: false

logging.level: debug
logging.to_files: true
logging.files:
path: "/logs/beats/heartbeat/"
name: heartbeat.log
keepfiles: 7

processors:

  • add_observer_metadata:

path.home: /opt/beats/7.4.0/heartbeat/
path.config: /opt/beats/7.4.0/heartbeat/
path.data: /opt/beats/7.4.0/heartbeat/data/
path.logs: /logs/beats/heartbeat/

4

I have added the logs and more details above

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.