Heartbeat showing my nodes as down but they are up

[marone (Marwane Chahoud)]

I set up heartbeat with my cluster ELK 7.12 (2 nodes) running on separate VMs, everything working fine, but when I enabled transport ssl and generated certificates for each node, heartbeat keeps showing that my nodes are down but they are up. Kibana is running on node1.

Here is heartbeat.yml for node1:

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: monitor-my-cluster-node1
  # Human readable display name for this service in Uptime UI and elsewhere
  name: Monitor node1
  # List or urls to query
  urls: ["http://localhost:9200"]
  # Configure task schedule
  schedule: '@every 10s'
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression

setup.kibana:
        host: "localhost:5601"
output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["node1:9200", "node2:9200"]
        username: "elastic"
        password: "${ES_PWD}"
processors:
  - add_observer_metadata:

and heartbeat.yml for node2:

heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s
# Configure monitors inline
heartbeat.monitors:
- type: http
  # ID used to uniquely identify this monitor in elasticsearch even if the config changes
  id: monitor-my-cluster-node2
  # Human readable display name for this service in Uptime UI and elsewhere
  name: Monitor node2
  # List or urls to query
  urls: ["http://localhost:9200"]
  # Configure task schedule
  schedule: '@every 10s'
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
        host: "node1:5601"
output.elasticsearch:
  # Array of hosts to connect to.
        hosts: ["node1:9200", "node2:9200"]
        username: "elastic"
        password: "${ES_PWD}"
processors:
  - add_observer_metadata:

checked heartbeat logs and showing no errors, what's wrong with my conf please?
Below a screenshot from kibana of heartbeat result:

[Jason_Slater]

If you expand the monitor which is displayed as "down" in the Uptime app, you may see the reason it's marked as down even though it's up.

From your heartbeat.yml configs, I see that you are checking "http://..." but recently added SSL certs. You might try changing these to https://... and adding elastic username/password as well if you've enabled authentication.

[marone (Marwane Chahoud)]

Thanks for your reply, I set up certificates for both nodes from here, I already specified username/password for elasticsearch in heartbeat.yml for both nodes. As you said I checked the error from the monitor showing me the following message:

Most recent error (a few seconds ago)
[401 Unauthorized]

for checking https instead of http, before I tried with curl as follow: curl -v -u myusername:mypwd https://localhost:9200 and the response is:

* Rebuilt URL to: https://localhost:9200/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9200 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* stopped the pause stream!
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

executing curl command with http instead of https, given the username/password is working

so I think the problem is not from http (correct me if I'm wrong), but I guess somehow heartbeat can't access elasticsearch given the username and password (I stored the password using the heartbeat keystore)!

[Jason_Slater]

According to a ticket I just read on the openssl github repo:

This error usually occurs if you attempt to connect to something that isn't using SSL/TLS

so perhaps http:// is correct in your case until you get https:// sorted out

re: the 401, I saw the output.elasticsearch credentials:

username: "elastic"
password: "${ES_PWD}"

but I didn't see username/password under heartbeat.monitors, at the same level as this:

  urls: ["http://localhost:9200"]

[marone (Marwane Chahoud)]

Thanks it's working , in my yml configuration there was no username/password under heartbeat.monitors, I checked the complete configuration from here.