Uptime heartbeat service is failing

Hi all,

I am trying observability uptime monitoring.
elastic .17.1
kibana .17.1
heartbeat 7.17.1

############################# Heartbeat ######################################

# Directory to load monitor definitions
heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  enabled: true
  id: s1
  name: s1
  urls: ["http://local_ip:9010/health-check"]
  schedule: '@every 10s'
  timeout: 30s  # Added timeout to avoid indefinite hangs

- type: http
  enabled: true
  id: s2
  name: s2
  urls: ["http://local_ip:9201/health-check"]
  schedule: '@every 10s'
  timeout: 30s  # Added timeout to avoid indefinite hangs

# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# =================================== Kibana ===================================
setup.kibana:
  host: "http://local_ip:5601"  # Explicitly added "http://"

# ================================== Outputs ===================================
output.elasticsearch:
  hosts: ["http://local_ip:9200"]  # Explicitly added "http://"

# ================================= Processors =================================
processors:
  - add_observer_metadata:
      geo:
        name: us-east-1a
        location: "37.926868, -78.024902"  # Example geo-location

# ================================== Logging ===================================
logging.level: info  # Default is 'info'; change to 'debug' if needed

# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true  # Set to true if you want to enable Stack Monitoring

error ->

× heartbeat-elastic.service - Ping remote services for availability and log results to Elasticsearch or send to Logstash.
     Loaded: loaded (/lib/systemd/system/heartbeat-elastic.service; disabled; vendor preset: enabled)
     Active: failed (Result: core-dump) since Thu 2025-01-23 13:34:01 IST; 1min 18s ago
       Docs: https://www.elastic.co/beats/heartbeat
    Process: 1389106 ExecStart=/usr/share/heartbeat/bin/heartbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=dumped, signal=ABRT)
   Main PID: 1389106 (code=dumped, signal=ABRT)
        CPU: 82ms

Jan 23 13:34:01 VCPL-LP-93 systemd[1]: heartbeat-elastic.service: Scheduled restart job, restart counter is at 29.
Jan 23 13:34:01 VCPL-LP-93 systemd[1]: Stopped Ping remote services for availability and log results to Elasticsearch or send to Logstash..
Jan 23 13:34:01 VCPL-LP-93 systemd[1]: heartbeat-elastic.service: Start request repeated too quickly.
Jan 23 13:34:01 VCPL-LP-93 systemd[1]: heartbeat-elastic.service: Failed with result 'core-dump'.
Jan 23 13:34:01 VCPL-LP-93 systemd[1]: Failed to start Ping remote services for availability and log results to Elasticsearch or send to Logstash..

  1. service is gets fail after few minutes
  2. status is updating but circular graph is not updating. plus it takes at least 1 min to update. how can i do it quickly. please see image below.

after doing so many times refresh also, its not working.

Thank you in advance

Can you provide details from the hearbeat log?

Hi

Thanks for quick reply.
can you tell me how to get that ?

Should be in /var/log/heartbeat, if is not set, add logging as it's recommended here.

2025-01-23T13:15:50.315+0530    INFO    instance/beat.go:686    Home path: [/usr/share/heartbeat] Config path: [/etc/heartbeat] Data path: [/var/lib/heartbeat] Logs path: [/var/log/heartbeat] Hostfs Path: [/]
2025-01-23T13:15:50.317+0530    INFO    instance/beat.go:694    Beat ID: fc3e5b8e-dad6-40df-b42c-d3049c89fe1b
2025-01-23T13:15:50.321+0530    INFO    [beat]  instance/beat.go:1040   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/heartbeat", "data": "/var/lib/heartbeat", "home": "/usr/share/heartbeat", "logs": "/var/log/heartbeat"}, "type": "heartbeat", "uuid": "fc3e5b8e-dad6-40df-b42c-d3049c89fe1b"}}}
2025-01-23T13:15:50.321+0530    INFO    [beat]  instance/beat.go:1049   Build info      {"system_info": {"build": {"commit": "1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff", "libbeat": "7.17.1", "time": "2022-02-23T23:44:00.000Z", "version": "7.17.1"}}}
2025-01-23T13:15:50.321+0530    INFO    [beat]  instance/beat.go:1052   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":16,"version":"go1.17.6"}}}
2025-01-23T13:15:50.323+0530    INFO    [beat]  instance/beat.go:1056   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2025-01-22T22:16:15+05:30","containerized":false,"name":"VCPL-LP-93","ip":["127.0.0.1/8","::1/128","10.40.13.27/23","fe80::8368:a1b1:33cb:63aa/64","172.22.0.1/16","172.24.0.1/16","172.27.0.1/16","172.26.0.1/16","172.28.0.1/16","172.19.0.1/16","fe80::42:5cff:fe23:26f/64","172.17.0.1/16","172.25.0.1/16","172.23.0.1/16","fe80::42:59ff:fec2:ef84/64","172.20.0.1/16","fe80::42:2ff:feb8:9b24/64","172.21.0.1/16","fe80::42:22ff:fe2a:93ec/64","172.18.0.1/16","fe80::42:27ff:fe3d:b21d/64","fe80::3887:fcff:fe2b:134d/64","fe80::f0f7:adff:fe12:dbed/64","fe80::cf0:e1ff:fe33:423d/64","fe80::90aa:3cff:feaf:a3f3/64","fe80::5494:a3ff:fe94:825b/64","fe80::7867:f2ff:fe9b:b19d/64","fe80::e803:2ff:fe28:fd93/64","fe80::74cd:90ff:fe46:5936/64","fe80::9c51:79ff:fea8:a01f/64","172.29.0.1/16","fe80::42:8aff:fe32:e295/64","fe80::bc0a:3cff:febf:cab6/64"],"kernel_version":"6.8.0-50-generic","mac":["64:6e:e0:65:10:ee","02:42:77:06:33:5c","02:42:5d:8d:0e:2a","02:42:79:a9:2d:7d","02:42:dc:e8:89:1d","02:42:35:75:83:4f","02:42:5c:23:02:6f","02:42:90:08:a0:96","02:42:ef:07:8c:bb","02:42:59:c2:ef:84","02:42:02:b8:9b:24","02:42:22:2a:93:ec","02:42:27:3d:b2:1d","3a:87:fc:2b:13:4d","f2:f7:ad:12:db:ed","0e:f0:e1:33:42:3d","92:aa:3c:af:a3:f3","56:94:a3:94:82:5b","7a:67:f2:9b:b1:9d","ea:03:02:28:fd:93","76:cd:90:46:59:36","9e:51:79:a8:a0:1f","02:42:8a:32:e2:95","be:0a:3c:bf:ca:b6"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.5 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":5,"codename":"jammy"},"timezone":"IST","timezone_offset_sec":19800,"id":"4368346214834eef8ca074b5ede63651"}}}
2025-01-23T13:15:50.324+0530    INFO    [beat]  instance/beat.go:1085   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["net_raw"],"effective":["net_raw"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/etc/heartbeat", "exe": "/usr/share/heartbeat/bin/heartbeat", "name": "heartbeat", "pid": 1237921, "ppid": 1237920, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2025-01-23T13:15:49.270+0530"}}}
2025-01-23T13:15:50.324+0530    INFO    instance/beat.go:328    Setup Beat: heartbeat; Version: 7.17.1
2025-01-23T13:15:50.324+0530    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'heartbeat-7.17.1' as ILM is enabled.
2025-01-23T13:15:50.324+0530    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: http://10.40.13.27:9200
2025-01-23T13:15:50.324+0530    INFO    [publisher]     pipeline/module.go:113  Beat name: VCPL-LP-93

There is no details there.
Check for more details in the journal log
journalctl -u heartbeat.service -n --since "2 hour ago"

Also might help to set logging.level: debug in heartbeat.yml

Fatal glibc error: rseq registration failed
i am trying different solutions from google, but not able to resolve. it would be great if you give me steps to how to install glibc.

A similar problem.

able to resolve above issue by adding code below in heartbeat.yml file

seccomp:
  default_action: allow 
  syscalls:
  - action: allow
    names:
    - rseq

now please help me to understand .. where is the setting to reduce refresh interval of that circular graph. it takes too much time to refresh/update.

and status section takes around 1 min to refresh/update.

Thank you in advance.

Not sure why is so slow.
If you think about how often will the monitor collect data, there is parameter: schedule: '@every 10s'

that i have already done. you can see that in my 1st message. but where is the setting to reduce refresh interval ?

If you are talking about the dashboard internal, then in Kibana.

i"ll check that. can u tell me how to add ssl configuration in heartbeat.yml file ??
heartbeat.yml file

############################# Heartbeat ######################################

# Directory to load monitor definitions
heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: true
  reload.period: 5s

# Configure monitors inline
heartbeat.monitors:
- type: http
  enabled: true
  id: S1
  name: S1
  urls: ["http://local_ip:9010/health-check"]
  schedule: '@every 10s'
  timeout: 30s  # Added timeout to avoid indefinite hangs
#  timeout to avoid indefinite hangs

# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

# =================================== Kibana ===================================
setup.kibana:
  host: "https://local_ip:5601"  # Explicitly added "http://"
  ssl:
    enabled: true  # Set to true if using SSL
    certificate_authorities: ["path_to_cert"]
    certificate: "path_to_cert"
    key: "path_to_cert"
    verification_mode: "certificate"

# ================================== Outputs ===================================
output.elasticsearch:
  hosts: ["https://local_ip:9200"]  # Explicitly added "http://"
  ssl:
    enabled: true  # Set to true if using SSL
    certificate_authorities: ["path_to_cert"]
    certificate: "path_to_cert"
    key: "path_to_cert"
    verification_mode: "certificate"

# ================================= Processors =================================
# processors:
#   - add_observer_metadata:
#       geo:
#         name: us-east-1a
#         location: "37.926868, -78.024902"  # Example geo-location

# ================================== Logging ===================================
logging.level: info  # Default is 'info'; change to 'debug' if needed

# ============================= X-Pack Monitoring ==============================
monitoring.enabled: true  # Set to true if you want to enable Stack Monitoring


seccomp:
  default_action: allow 
  syscalls:
  - action: allow
    names:
    - rseq

even if adding correct certificate i am getting below error

Jan 24 18:24:59 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:24:59.480+0530        ERROR        [esclientleg]        transport/logging.go:37        Error dialing x509: certificate signed by unknown author>
Jan 24 18:25:13 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:13.397+0530        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s        {"monitoring": {"metrics": >
Jan 24 18:25:14 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:14.199+0530        ERROR        [esclientleg]        transport/logging.go:37        Error dialing x509: certificate signed by unknown author>
Jan 24 18:25:43 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:43.389+0530        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s        {"monitoring": {"metrics": >
Jan 24 18:25:44 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:44.212+0530        ERROR        [esclientleg]        transport/logging.go:37        Error dialing x509: certificate signed by unknown author>
Jan 24 18:25:48 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:48.596+0530        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(elasticsearch(>
Jan 24 18:25:48 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:48.596+0530        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(elasticse>
Jan 24 18:25:48 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:48.596+0530        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer
Jan 24 18:25:48 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:48.597+0530        INFO        [publisher]        pipeline/retry.go:223          done
Jan 24 18:25:48 VCPL-LP-93 heartbeat[380760]: 2025-01-24T18:25:48.606+0530        ERROR        [esclientleg]        transport/logging.go:37        Error dialing x509: certificate signed by unknown author>

thank you in advance.

Something is not OK with the certificate, it's not the same author/issuer. Pay attention which cert you put in which param. You have 3: CA, cert, private key.

Temporarily you can set verification_mode: "none" . In case you need the documentation.